We're Hiring!

Mattermost, Inc.

Upgrade MM 5.3.1 to latest version fails

Hello ahmaddanial,
ah, there is some work to do for me, great :wink:
I will try it soon and get back to you.
Thank you very much.

1 Like

Hello ahmaddanial,

first let me say to the content/settings of mattermost.service:
NO, that was not intended in this manner. I guess I read that somewhere as a workaround to my previous failed update attempts before I wrote in this forum.

Well, I’ve done as written by you above – what I already knew :wink:

Below that’s now my new part:

remove mattermost.service -> I’ve found two of them, so I removed both

rm /etc/systemd/system/multi-user.target.wants/mattermost.service
rm /lib/systemd/system/mattermost.service

cd /opt/mattermost
sudo -u mattermost ./bin/mattermost
{“level”:“info”,“ts”:1610571078.3621776,“caller”:“utils/i18n.go:83”,“msg”:“Loaded system translations”,“for locale”:“de”,“from locale”:"/opt/mattermost/i18n/de.json"}
{“level”:“info”,“ts”:1610571078.3624952,“caller”:“app/server_app_adapters.go:58”,“msg”:“Server is initializing…”}
{“level”:“info”,“ts”:1610571078.364868,“caller”:“sqlstore/supplier.go:227”,“msg”:“Pinging SQL”,“database”:“master”}
{“level”:“error”,“ts”:1610571078.418023,“caller”:“sqlstore/supplier.go:915”,“msg”:“Failed to create index”,“error”:“Error 1709: Index column size too large. The maximum column size is 767 bytes.”}

now there are no mattermost.service files available

I created mattermost.service as described in DOCS 10b

vi /lib/systemd/system/mattermost.service

1 [Unit]
2 Description=Mattermost
3 After=network.target
4 After=mysql.service
5 BindsTo=mysql.service
6
7 [Service]
8 Type=notify
9 ExecStart=/opt/mattermost/bin/mattermost
10 TimeoutStartSec=3600
11 Restart=always
12 RestartSec=10
13 WorkingDirectory=/opt/mattermost
14 User=mattermost
15 Group=mattermost
16 LimitNOFILE=49152
17
18 [Install]
19 WantedBy=mysql.service

systemctl start mattermost
Failed to start mattermost.service: Unit mattermost.service not found.

I guess: copy mattermost.service to second location (see above)

cp /lib/systemd/system/mattermost.service /etc/systemd/system/multi-user.target.wants

but the same issue

systemctl start mattermost
Failed to start mattermost.service: Unit mattermost.service not found.

systemctl status mattermost
● mattermost.service
Loaded: not-found (Reason: No such file or directory)
Active: failed (Result: resources) since Mit 2021-01-13 21:57:22 CET; 40min ago
Main PID: 842 (code=exited, status=119)

Jän 13 21:57:15 VIEASS3 systemd[1]: mattermost.service: Unit entered failed state.
Jän 13 21:57:15 VIEASS3 systemd[1]: mattermost.service: Failed with result ‘exit-code’.
Jän 13 21:57:22 VIEASS3 systemd[1]: mattermost.service: Service hold-off time over, scheduling restart.
Jän 13 21:57:22 VIEASS3 systemd[1]: mattermost.service: Failed to schedule restart job: Unit mattermost.service not found.
Jän 13 21:57:22 VIEASS3 systemd[1]: mattermost.service: Unit entered failed state.
Jän 13 21:57:22 VIEASS3 systemd[1]: mattermost.service: Failed with result ‘resources’.

I hope you can see any issue. But it may also be that I have misunderstood something.
Let me know how I can further assist.

Hello, @SuziQ

No worries. Moving back to starting Mattermost manually without using the mattermost.service, can you please remove any existing systemd file(s) tied to Mattermost and start Mattermost manually using the command I asked you earlier within /opt/mattermost/:

sudo -u mattermost ./bin/mattermost

Hello ahmaddanial,

I’m sorry, which are the existing systemd file(s) tied to Mattermost?
You mean

  • /lib/systemd/system/mattermost.service and
  • /etc/systemd/system/multi-user.target.wants/mattermost.service?

Ok, I’ve tried it and have removed this two files.
But where’s my MM config now?

If there’s no mattermost.service needed what should happen after “sudo -u mattermost ./bin/mattermost”? What should I try? What should I get or see?

sudo -u mattermost ./bin/mattermost

{“level”:“info”,“ts”:1610657383.0816736,“caller”:“utils/i18n.go:83”,“msg”:“Loaded system translations”,“for locale”:“de”,“from locale”:"/opt/mattermost/i18n/de.json"}
{“level”:“info”,“ts”:1610657383.0973482,“caller”:“app/server_app_adapters.go:58”,“msg”:“Server is initializing…”}
{“level”:“info”,“ts”:1610657383.2245412,“caller”:“sqlstore/supplier.go:227”,“msg”:“Pinging SQL”,“database”:“master”}
{“level”:“error”,“ts”:1610657383.6669023,“caller”:“sqlstore/supplier.go:915”,“msg”:“Failed to create index”,“error”:“Error 1709: Index column size too large. The maximum column size is 767 bytes.”}

I’m a little bit confused about what you plan and what should happen or what should I try or get or see!

How big is your Mattermost database, if I can ask? For example, how many users and total messages do you have?

According to phpMyAdmin, the database is 15.6 MiB in size, 7-8 teams, about 4,000 messages. These are clubs that have deposited their protocols and agreements, their strategies for the future here.
Why do you ask?

Is there a way to convert the database from old Mattermost version to the new one?
What says this error message > “Failed to create index”,“error”:“Error 1709: Index column size too large. The maximum column size is 767 bytes.”

Within a major version (v5) I would not have thought there would be such problems updating. But I go along with everything so that I can get the clubs their data. I just need to know WHAT to do …

Hey @SuziQ, don’t worry, we’re going to get this figured out. I think I may have found a solution, but before I tell you to do anything to your database, I just want to make sure I have the right information from you, so that I don’t inadvertently have you do the wrong thing.

So, to be safe, can you confirm the Database and Engine that you are using?
An example would be MySQL running with MariaDB

Also, and more importantly, could you provide the schema of the table OutgoingWebhooks in your Mattermost database?
This is what mine looks like, I’m running Mattermost 5.28.1. I will be able to see based on what you provide, if the issue I suspect, which is incorrect database character type on upgrade, is the actual issue. This is my schema for the OutgoingWebhooks table:

mysql> describe OutgoingWebhooks;
+--------------+--------------+------+-----+---------+-------+
| Field        | Type         | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+-------+
| Id           | varchar(26)  | NO   | PRI | NULL    |       |
| Token        | varchar(26)  | YES  |     | NULL    |       |
| CreateAt     | bigint       | YES  | MUL | NULL    |       |
| UpdateAt     | bigint       | YES  | MUL | NULL    |       |
| DeleteAt     | bigint       | YES  | MUL | NULL    |       |
| CreatorId    | varchar(26)  | YES  |     | NULL    |       |
| ChannelId    | varchar(26)  | YES  |     | NULL    |       |
| TeamId       | varchar(26)  | YES  | MUL | NULL    |       |
| TriggerWords | text         | YES  |     | NULL    |       |
| TriggerWhen  | int          | YES  |     | NULL    |       |
| CallbackURLs | text         | YES  |     | NULL    |       |
| DisplayName  | varchar(64)  | YES  |     | NULL    |       |
| Description  | text         | YES  |     | NULL    |       |
| ContentType  | varchar(128) | YES  |     | NULL    |       |
| Username     | varchar(64)  | YES  |     | NULL    |       |
| IconURL      | text         | YES  |     | NULL    |       |
+--------------+--------------+------+-----+---------+-------+
16 rows in set (0.01 sec)

Thank you! If you encounter any troubles getting your schema, it’s quite alright, just let me know and I will be glad to help you out!

Hello @XxLilBoPeepsxX,

thank you for your help and also for your confidence. (y)

current version MySQL running with MariaDB

mysql -V
mysql Ver 15.1 Distrib 10.0.38-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

my schema for the OutgoingWebhooks table

mysql -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 548
Server version: 10.0.38-MariaDB-0ubuntu0.16.04.1 Ubuntu 16.04

Copyright © 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

MariaDB [(none)]>
MariaDB [(none)]> use mattermost;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [mattermost]> describe OutgoingWebhooks;

+--------------+--------------+------+-----+---------+-------+
| Field        | Type         | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+-------+
| Id           | varchar(26)  | NO   | PRI | NULL    |       |
| Token        | varchar(26)  | YES  |     | NULL    |       |
| CreateAt     | bigint(20)   | YES  | MUL | NULL    |       |
| UpdateAt     | bigint(20)   | YES  | MUL | NULL    |       |
| DeleteAt     | bigint(20)   | YES  | MUL | NULL    |       |
| CreatorId    | varchar(26)  | YES  |     | NULL    |       |
| ChannelId    | varchar(26)  | YES  |     | NULL    |       |
| TeamId       | varchar(26)  | YES  | MUL | NULL    |       |
| TriggerWords | text         | YES  |     | NULL    |       |
| TriggerWhen  | int(11)      | YES  |     | NULL    |       |
| CallbackURLs | text         | YES  |     | NULL    |       |
| DisplayName  | varchar(64)  | YES  |     | NULL    |       |
| Description  | text         | YES  |     | NULL    |       |
| ContentType  | varchar(128) | YES  |     | NULL    |       |
| Username     | varchar(64)  | YES  |     | NULL    |       |
| IconURL      | text         | YES  |     | NULL    |       |
+--------------+--------------+------+-----+---------+-------+
16 rows in set (0.00 sec)
1 Like

Alright so it looks like your database schema is correct, that eliminates the first of two potential issues. The first one I referenced a Github issue that has already been closed for Mattermost from, the second has to do with the default correlation and reading of the database as a whole, which if not set the correct way, makes it so Mattermost cannot function properly.

By any chance, could you provide the contents of your my.cnf MySQL configuration file?

My first thought was also a faulty database. However, where should this error come from? Mattermost v5.3 is running correctly, as you know.

Here is my /etc/mysql/my.cnf:

cat /etc/mysql/my.cnf

> # This file has been automatically moved from your previous
> # /etc/mysql/my.cnf, with just this comment added at the top, to maintain MySQL
> # operation using your previously customised configuration.
> 
> # To switch to the new packaging configuration for automated management of
> # /etc/mysql/my.cnf across multiple variants:
> #
> # 1. Move your customisations from this file to /etc/mysql/conf.d/ and
> #    to /etc/mysql/<variant>.conf.d/ as appropriate.
> # 2. Run "update-alternatives --remove my.cnf /etc/mysql/my.cnf.migrated"
> # 3. Remove the file /etc/mysql/my.cnf.migrated
> 
> # MariaDB database server configuration file.
> #
> # You can copy this file to one of:
> # - "/etc/mysql/my.cnf" to set global options,
> # - "~/.my.cnf" to set user-specific options.
> # 
> # One can use all long options that the program supports.
> # Run program with --help to get a list of available options and with
> # --print-defaults to see which it would actually understand and use.
> #
> # For explanations see
> # http://dev.mysql.com/doc/mysql/en/server-system-variables.html
> 
> # This will be passed to all mysql clients
> # It has been reported that passwords should be enclosed with ticks/quotes
> # escpecially if they contain "#" chars...
> # Remember to edit /etc/mysql/debian.cnf when changing the socket location.
> [client]
> port		= 3306
> socket		= /var/run/mysqld/mysqld.sock
> 
> # Here is entries for some specific programs
> # The following values assume you have at least 32M ram
> 
> # This was formally known as [safe_mysqld]. Both versions are currently parsed.
> [mysqld_safe]
> socket		= /var/run/mysqld/mysqld.sock
> nice		= 0
> 
> [mysqld]
> ### 8.Nov-16: Added by Gruppe-ASS
> ### See also: https://dba.stackexchange.com/questions/8982/what-is-the-best-way-to-reduce-the-size-of-ibdata-in-mysql
> innodb_file_per_table  = 1
> ### innodb_flush_method        = 0_DIRECT
> ### innodb_log_file_size       = 2G
> ### innodb_buffer_pool_size    = 8G
> 
> # needed by NC 15.0.7+
> innodb_large_prefix = true
> innodb_file_format = Barracuda
> 
> #
> # * Basic Settings
> #
> user		= mysql
> pid-file	= /var/run/mysqld/mysqld.pid
> socket		= /var/run/mysqld/mysqld.sock
> port		= 3306
> basedir		= /usr
> datadir		= /var/lib/mysql
> tmpdir		= /tmp
> lc_messages_dir	= /usr/share/mysql
> lc_messages	= en_US
> skip-external-locking
> #
> # Instead of skip-networking the default is now to listen only on
> # localhost which is more compatible and is not less secure.
> ### ASS: bind-address		= 127.0.0.1
> bind-address		= 0.0.0.0
> #
> # * Fine Tuning
> #
> max_connections		= 100
> connect_timeout		= 5
> wait_timeout		= 600
> max_allowed_packet	= 16M
> thread_cache_size       = 128
> sort_buffer_size	= 4M
> bulk_insert_buffer_size	= 16M
> tmp_table_size		= 32M
> max_heap_table_size	= 32M
> #
> # * MyISAM
> #
> # This replaces the startup script and checks MyISAM tables if needed
> # the first time they are touched. On error, make copy and try a repair.
> myisam_recover          = BACKUP
> key_buffer_size		= 128M
> #open-files-limit	= 2000
> table_open_cache	= 400
> myisam_sort_buffer_size	= 512M
> concurrent_insert	= 2
> read_buffer_size	= 2M
> read_rnd_buffer_size	= 1M
> #
> # * Query Cache Configuration
> #
> # Cache only tiny result sets, so we can fit more in the query cache.
> query_cache_limit		= 128K
> query_cache_size		= 64M
> # for more write intensive setups, set to DEMAND or OFF
> #query_cache_type		= DEMAND
> #
> # * Logging and Replication
> #
> # Both location gets rotated by the cronjob.
> # Be aware that this log type is a performance killer.
> # As of 5.1 you can enable the log at runtime!
> #general_log_file        = /var/log/mysql/mysql.log
> #general_log             = 1
> #
> # Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf.
> #
> # we do want to know about network errors and such
> log_warnings		= 2
> #
> # Enable the slow query log to see queries with especially long duration
> #slow_query_log[={0|1}]
> slow_query_log_file	= /var/log/mysql/mariadb-slow.log
> long_query_time = 10
> #log_slow_rate_limit	= 1000
> log_slow_verbosity	= query_plan
> 
> #log-queries-not-using-indexes
> #log_slow_admin_statements
> #
> # The following can be used as easy to replay backup logs or for replication.
> # note: if you are setting up a replication slave, see README.Debian about
> #       other settings you may need to change.
> #server-id		= 1
> #report_host		= master1
> #auto_increment_increment = 2
> #auto_increment_offset	= 1
> log_bin			= /var/log/mysql/mariadb-bin
> log_bin_index		= /var/log/mysql/mariadb-bin.index
> # not fab for performance, but safer
> #sync_binlog		= 1
> expire_logs_days	= 10
> max_binlog_size         = 100M
> # slaves
> #relay_log		= /var/log/mysql/relay-bin
> #relay_log_index	= /var/log/mysql/relay-bin.index
> #relay_log_info_file	= /var/log/mysql/relay-bin.info
> #log_slave_updates
> #read_only
> #
> # If applications support it, this stricter sql_mode prevents some
> # mistakes like inserting invalid dates etc.
> #sql_mode		= NO_ENGINE_SUBSTITUTION,TRADITIONAL
> #
> # * InnoDB
> #
> # InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
> # Read the manual for more InnoDB related options. There are many!
> default_storage_engine	= InnoDB
> # you can't just change log file size, requires special procedure
> #innodb_log_file_size	= 50M
> innodb_buffer_pool_size	= 256M
> innodb_log_buffer_size	= 8M
> innodb_file_per_table	= 1
> innodb_open_files	= 400
> innodb_io_capacity	= 400
> innodb_flush_method	= O_DIRECT
> #
> # * Security Features
> #
> # Read the manual, too, if you want chroot!
> # chroot = /var/lib/mysql/
> #
> # For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
> #
> # ssl-ca=/etc/mysql/cacert.pem
> # ssl-cert=/etc/mysql/server-cert.pem
> # ssl-key=/etc/mysql/server-key.pem
> 
> 
> 
> [mysqldump]
> quick
> quote-names
> max_allowed_packet	= 16M
> 
> [mysql]
> #no-auto-rehash	# faster start of mysql but no tab completition
> 
> [isamchk]
> key_buffer		= 16M
> 
> #
> # * IMPORTANT: Additional settings that can override those from this file!
> #   The files must end with '.cnf', otherwise they'll be ignored.
> #
> !includedir /etc/mysql/conf.d/

It would come from the configuration of the Database formatting, schema, etc that, if incorrect, would hinder the proper operations of Mattermost even while it is running. I have been referring to this Mattermost Github issue, which seems to be exactly what you are experiencing. If you read it from top-down, you will see me asking about the schema of the table is related to the first posted possible solution in the Github thread. At this point in time, I am now asking about this possible solution, which from what I can see, is what your configuration is set to be:

cedarice commented on Feb 28, 2019
I found a solution:
For mariadb, update your *my.cnf file with following configuration,

innodb_default_row_format=dynamic
innodb_file_format=barracuda
innodb_file_per_table=true
innodb_large_prefix=true

Then, you have to restart mariadb service for updated configuration to take effect.

reference: https://stackoverflow.com/a/52778785

If you take a look at that, could you let me know what you find, from the various schemas that you may want to check, as well as from your configuration files?

I’m sorry, but at the moment I have no idea what to do, what to find, what schemas are meant.

Your my.cnf entries are all there except:
“innodb_default_row_format=dynamic”.

I added this line and can’t restart MariaDB now.

systemctl status mysql.service

● mysql.service - LSB: Start and stop the mysql database server daemon
Loaded: loaded (/etc/init.d/mysql; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Die 2021-01-19 19:31:09 CET; 39s ago
Docs: man:systemd-sysv-generator(8)
Process: 7469 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
Process: 7509 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)

Jän 19 19:30:38 SERVER systemd[1]: Starting LSB: Start and stop the mysql database server daemon…
Jän 19 19:30:38 SERVER mysql[7509]: * Starting MariaDB database server mysqld
Jän 19 19:31:09 SERVER mysql[7509]: …fail!
Jän 19 19:31:09 SERVER systemd[1]: mysql.service: Control process exited, code=exited status=1
Jän 19 19:31:09 SERVER systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
Jän 19 19:31:09 SERVER systemd[1]: mysql.service: Unit entered failed state.
Jän 19 19:31:09 SERVER systemd[1]: mysql.service: Failed with result ‘exit-code’.

I also have a number of websites running on Apache, including Nextcloud for these clubs, which are created in Mattermost.

And what other configuration files do you mean?

I found as follows:

updatedb; locate my.cnf

/etc/alternatives/my.cnf
/etc/mysql/my.cnf
/etc/mysql/my.cnf.dpkg-old
/etc/mysql/my.cnf.fallback
/etc/mysql/my.cnf.migrated
/root/.my.cnf
/var/lib/dpkg/alternatives/my.cnf

locate mariadb.cnf

/etc/mysql/mariadb.cnf
/etc/mysql/conf.d/mariadb.cnf

If these files are of interest, I’ll post their contents here.

I’m sorry, I need more specific instructions on what to try.