Mattermost Peer-to-Peer Forum

How to start the server without pinging the DB


#1

Can I start the mattermost server without pinging the DB first?

I am using the latest version of mattermost server installed in an Amazon Linux EC2 instance and an AWS Aurora (MySQL 5.6) Database.

I start the server using the command:
sudo -u mattermost ./bin/mattermost

Mattermost first pings the IP indicated in the "datasource"section of file /opt/mattermost/config/config.json.

Is there an easy way to configure Mattermost server to NOT ping the DB first (maybe use telnet or something else to check that it is alive), connect to the DB and start the server?

Thank you.


Run Mattermost using AWS Aurora serverless
#2

@Miguel, the “ping” here is not an ICMP echo request, but the invocation of PingContext in the database/sql Go package. It’s just a way to verify connectivity to the database instead of waiting for the first use of the database.

Are you running into a problem as such?


#3

Hi Jesse,

Thank you very much for the response.

Yes. The problem I am having is that I want to use the serverless Aurora DB service but the end point of the DB can’t be pinged. I asked AWS support to see if it can be changed and they said no.

This is the error I am getting when starting the server:

{“level”:“info”,“ts”:1551280409.4031305,“caller”:“app/app.go:143”,“msg”:“Server is initializing…”}
{“level”:“info”,“ts”:1551280409.405245,“caller”:“sqlstore/supplier.go:207”,“msg”:“Pinging SQL master database”}
[mysql] 2019/02/27 15:13:59 packets.go:36: read tcp x.x.x.x:36150->x.x.x.x:3306: i/o timeout
{“level”:“error”,“ts”:1551280439.434761,“caller”:“sqlstore/supplier.go:219”,“msg”:“Failed to ping DB retrying in 10 seconds err=context deadline exceeded”}

However, I can connect no problem to the DB using:
mysql — user=[user] --password -h [endpoint of cluster])

Thank you for your help!
Miguel


#4

Hmm, that error is just the server failing to connect to the Aurora DB service – it’s not necessary to be able to “ping” the database endpoint in an ICMP sense. (We’re just using the term in the logs because it’s illustrative of a connectivity test.)

Are you running that mysql invocation on the same box as the server in question? I’d suggest trying to narrow down connectivity issues between the box running Mattermost and the Aurora DB endpoint in question.

It’s also possible that there’s some kind of incompatibility between database/sql (and https://github.com/go-sql-driver/mysql) and Aurora’s serverless infrastructure, but it seems unlikely (otherwise it would break the mysql protocol).


#5

The MySQL DB is running on a cluster of instances that I do not have root access to. This DB cluster is “created” for me when I create an Aurora DB and I tell it to be in serverless mode.

There doesn’t seem to be any problems of connectivity though. I am running the mysql command from the EC2 instance the Mattermost server is installed on. I also can telnet on port 3306 from this EC2 to the DB endpoint.

Miguel


#6

Hi @Miguel,

Feedback from a community member:

In case you are allowed to share, it would be nice to read what exactly the Amazon support said. Is there a reason for them to use an IP address instead of a hostname?


#7

AWS does give me a hostname as the DB endpoint, and I have used both the hostname and the IP address in the "datasource"section of config.json, and both fail.
I can connect successfully to the DB using both the hostname and the IP address using a mysql command.

AWS tech support says that from the beginning of the service “pinging” the endpoint has never been allowed for security reasons. The weird thing is that I have read posts of people successfully connecting Mattermost to an AWS (RDS) MySQL Database, so I don’t understand how people did it. Maybe in previous versions Mattermost server was allowed to start without pinging first??. Can I change some setting so that Mattermost doesn’t check first and just connects to the DB?

Thank you for any help.


#8

Hi, @Miguel

In regards to the statement that you have here:

The weird thing is that I have read posts of people successfully connecting Mattermost to an AWS (RDS) MySQL Database, so I don’t understand how people did it.

Perhaps you can share the article(s) that you came across highlighting the success of doing this so we can dig for more information that we can utilize to make this work. Potentially, we can also try to get in touch with the users who successfully make it work to compare the configuration that they have with yours too to see if it is something missing from Mattermost or rather Aurora’s side instead.

If you don’t mind me asking, can you also share the snippet of the SQL settings from your config.json file as well just to ensure that we don’t miss out on anything minor?
"SqlSettings": {
"DriverName": "mysql",
"DataSource": "xxx:xxx@tcp(xxx:3306)/xxx?charset=utf8mb4,utf8\u0026readTimeout=30s\u0026writeTimeout=30s",},`


#9

As far as post from people who have done it, here you go:

The most intriguing is this “official” doc from Mattermost itself. The table right at the very end of the page indicates that I can use Aurora RDS for MySQL:

https://docs.mattermost.com/deployment/scaling.html

This person is using a RDS MySQL DB

Another example:

As far as SQL settings (password and key edited):

“SqlSettings”: {
“DriverName”: “mysql”,
“DataSource”: “mmuser: password @tcp(172.16.3.114:3306)/mattermosm?charset=utf8mb4,utf8\u0026readTimeout=30s\u0026writeTimeout=30s”,
“DataSourceReplicas”: [],
“DataSourceSearchReplicas”: [],
“MaxIdleConns”: 20,
“ConnMaxLifetimeMilliseconds”: 3600000,
“MaxOpenConns”: 300,
“Trace”: false,
“AtRestEncryptKey”: “key”,
“QueryTimeout”: 30
},


#10

Hi, @Miguel

Indeed, the table includes the information of Aurora RDS for both MySQL and PostgreSQL too. I went through the SQL settings you provided and while the settings look similar (except for the spaces in between), I wonder if the name mattermosm (instead of mattermost) is intended or just a small typo on your end under the DataSource object:

“DataSource”: “mmuser: **password** @tcp(172.16.3.114:3306)/mattermosm?charset=utf8mb4,utf8\u0026readTimeout=30s\u0026writeTimeout=30s”,

Apart from that, everything looks similar to mine. @jtor, perhaps you can share a little bit more insight on how Aurora is being used on your instance?