Mattermost, Inc.

Panic: Failed to create index pq: coluna "teamid" does not exist

After upgrading from version 8.1.2 for 8.2.2 (yum update gitlab-c) the following message appears in mattermost logs.

`2015-12-04_19:23:11.47249 panic: Failed to create index pq: coluna “teamid” não existe
2015-12-04_19:23:11.47257
2015-12-04_19:23:11.47259 goroutine 1 [running]:
2015-12-04_19:23:11.47259 github.com/mattermost/platform/store.SqlStore.createIndexIfNotExists(0xc20800ca80, 0xc20802e098, 0x1, 0x1, 0x7f69e882cdd0, 0xc20802e0b0, 0x7f69e882ce38, 0xc20802e0e8, 0x7f69e882cf08, 0xc20802e140, …)
2015-12-04_19:23:11.47262 /home/travis/gopath/src/github.com/mattermost/platform/store/sql_store.go:419 +0x57e
2015-12-04_19:23:11.47263 github.com/mattermost/platform/store.SqlStore.CreateIndexIfNotExists(0xc20800ca80, 0xc20802e098, 0x1, 0x1, 0x7f69e882cdd0, 0xc20802e0b0, 0x7f69e882ce38, 0xc20802e0e8, 0x7f69e882cf08, 0xc20802e140, …)
2015-12-04_19:23:11.47265 /home/travis/gopath/src/github.com/mattermost/platform/store/sql_store.go:392 +0xb4
2015-12-04_19:23:11.47266 github.com/mattermost/platform/store.SqlUserStore.CreateIndexesIfNotExists(0xc2080a4000)
2015-12-04_19:23:11.47267 /home/travis/gopath/src/github.com/mattermost/platform/store/sql_user_store.go:49 +0x90
2015-12-04_19:23:11.47268 github.com/mattermost/platform/store.NewSqlStore(0x0, 0x0)
2015-12-04_19:23:11.47273 /home/travis/gopath/src/github.com/mattermost/platform/store/sql_store.go:143 +0x119f
2015-12-04_19:23:11.47274 github.com/mattermost/platform/api.NewServer()
2015-12-04_19:23:11.47275 /home/travis/gopath/src/github.com/mattermost/platform/api/server.go:33 +0x15a
2015-12-04_19:23:11.47276 main.main()
2015-12-04_19:23:11.47277 /home/travis/gopath/src/github.com/mattermost/platform/mattermost.go:55 +0x4d8
2015-12-04_19:23:11.47280
2015-12-04_19:23:11.47281 goroutine 9 [chan receive]:
2015-12-04_19:23:11.47282 database/sql.(*DB).connectionOpener(0xc208055900)
2015-12-04_19:23:11.47283 /usr/local/go/src/database/sql/sql.go:589 +0x4c
2015-12-04_19:23:11.47284 created by database/sql.Open
2015-12-04_19:23:11.47285 /usr/local/go/src/database/sql/sql.go:452 +0x31c
2015-12-04_19:23:11.47286
2015-12-04_19:23:11.47287 goroutine 6 [syscall]:
2015-12-04_19:23:11.47288 os/signal.loop()
2015-12-04_19:23:11.47289 /usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
2015-12-04_19:23:11.47290 created by os/signal.init·1
2015-12-04_19:23:11.47291 /usr/local/go/src/os/signal/signal_unix.go:27 +0x35
2015-12-04_19:23:11.47292
2015-12-04_19:23:11.47293 goroutine 7 [select]:
2015-12-04_19:23:11.47294 code.google.com/p/log4go.func·002()
2015-12-04_19:23:11.47295 /home/travis/gopath/src/github.com/mattermost/platform/Godeps/_workspace/src/code.google.com/p/log4go/filelog.go:84 +0xa53
2015-12-04_19:23:11.47296 created by code.google.com/p/log4go.NewFileLogWriter
2015-12-04_19:23:11.47297 /home/travis/gopath/src/github.com/mattermost/platform/Godeps/_workspace/src/code.google.com/p/log4go/filelog.go:116 +0x3da
2015-12-04_19:23:11.47298
2015-12-04_19:23:11.47299 goroutine 8 [chan receive]:
2015-12-04_19:23:11.47300 database/sql.(*DB).connectionOpener(0xc208055720)
2015-12-04_19:23:11.47301 /usr/local/go/src/database/sql/sql.go:589 +0x4c
2015-12-04_19:23:11.47307 created by database/sql.Open
2015-12-04_19:23:11.47308 /usr/local/go/src/database/sql/sql.go:452 +0x31c
2015-12-04_19:23:11.51272 sql-trace:17:23:11.512610 SELECT Value FROM Systems WHERE Name=‘Version’ [] (3.222112ms)
2015-12-04_19:23:11.51517 sql-trace:17:23:11.515102 SELECT count(relname) FROM pg_class WHERE relname=$1 [1:“sessions”] (2.420948ms)
2015-12-04_19:23:11.51781 sql-trace:17:23:11.517745 SELECT COUNT(0)
2015-12-04_19:23:11.51785 FROM pg_attribute
2015-12-04_19:23:11.51786 WHERE attrelid = $1::regclass
2015-12-04_19:23:11.51787 AND attname = $2
2015-12-04_19:23:11.51788 AND NOT attisdropped [1:“sessions” 2:“altid”] (2.570419ms)
2015-12-04_19:23:11.51955 sql-trace:17:23:11.519469 create table if not exists “teams” (“id” varchar(26) not null primary key, “createat” bigint, “updateat” bigint, “deleteat” bigint, “displayname” varchar(64), “name” varchar(64) unique, “email” varchar(128), “type” text, “companyname” varchar(64), “alloweddomains” varchar(500), “inviteid” varchar(32), “allowopeninvite” boolean, “allowteamlisting” boolean) ; [] (807.398µs)
2015-12-04_19:23:11.52021 sql-trace:17:23:11.520166 create table if not exists “channels” (“id” varchar(26) not null primary key, “createat” bigint, “updateat” bigint, “deleteat” bigint, “teamid” varchar(26), “type” varchar(1), “displayname” varchar(64), “name” varchar(64), “header” varchar(1024), “purpose” varchar(128), “lastpostat” bigint, “totalmsgcount” bigint, “extraupdateat” bigint, “creatorid” varchar(26), unique (“name”, “teamid”)) ; [] (585.851µs)
2015-12-04_19:23:11.52073 sql-trace:17:23:11.520699 create table if not exists “channelmembers” (“channelid” varchar(26) not null, “userid” varchar(26) not null, “roles” varchar(64), “lastviewedat” bigint, “msgcount” bigint, “mentioncount” bigint, “notifyprops” varchar(2000), “lastupdateat” bigint, primary key (“channelid”, “userid”)) ; [] (443.893µs)
2015-12-04_19:23:11.52182 sql-trace:17:23:11.521772 create table if not exists “posts” (“id” varchar(26) not null primary key, “createat” bigint, “updateat” bigint, “deleteat” bigint, “userid” varchar(26), “channelid” varchar(26), “rootid” varchar(26), “parentid” varchar(26), “originalid” text, “message” varchar(4000), “imgcount” bigint, “type” varchar(26), “props” text, “hashtags” varchar(1000), “filenames” varchar(4000)) ; [] (946.305µs)
2015-12-04_19:23:11.52305 sql-trace:17:23:11.522998 create table if not exists “users” (“id” varchar(26) not null primary key, “createat” bigint, “updateat” bigint, “deleteat” bigint, “teamid” varchar(26), “username” varchar(64), “password” varchar(128), “authdata” varchar(128), “authservice” varchar(32), “email” varchar(128), “emailverified” boolean, “nickname” varchar(64), “firstname” varchar(64), “lastname” varchar(64), “roles” varchar(64), “lastactivityat” bigint, “lastpingat” bigint, “allowmarketing” boolean, “props” varchar(4000), “notifyprops” varchar(2000), “themeprops” varchar(2000), “lastpasswordupdate” bigint, “lastpictureupdate” bigint, “failedattempts” integer, unique (“email”, “teamid”), unique (“username”, “teamid”)) ; [] (975.614µs)
2015-12-04_19:23:11.52401 sql-trace:17:23:11.523954 create table if not exists “audits” (“id” varchar(26) not null primary key, “createat” bigint, “userid” varchar(26), “action” varchar(64), “extrainfo” varchar(128), “ipaddress” varchar(64), “sessionid” varchar(26)) ; [] (850.991µs)
2015-12-04_19:23:11.52460 sql-trace:17:23:11.524552 create table if not exists “sessions” (“id” varchar(26) not null primary key, “token” varchar(26), “createat” bigint, “expiresat” bigint, “lastactivityat” bigint, “userid” varchar(26), “teamid” varchar(26), “deviceid” varchar(128), “roles” varchar(64), “isoauth” boolean, “props” varchar(1000)) ; [] (497.294µs)
2015-12-04_19:23:11.52572 sql-trace:17:23:11.525642 create table if not exists “oauthapps” (“id” varchar(26) not null primary key, “creatorid” varchar(26), “createat” bigint, “updateat” bigint, “clientsecret” varchar(128), “name” varchar(64), “description” varchar(512), “callbackurls” varchar(1024), “homepage” varchar(256)) ; [] (1.000052ms)
2015-12-04_19:23:11.52675 sql-trace:17:23:11.526697 create table if not exists “oauthauthdata” (“clientid” varchar(26), “userid” varchar(26), “code” varchar(128) not null primary key, “expiresin” integer, “createat” bigint, “redirecturi” varchar(256), “state” varchar(128), “scope” varchar(128)) ; [] (965.982µs)
2015-12-04_19:23:11.52783 sql-trace:17:23:11.527773 create table if not exists “oauthaccessdata” (“authcode” varchar(128), “token” varchar(26) not null primary key, “refreshtoken” varchar(26), “redirecturi” varchar(256)) ; [] (1.004912ms)
2015-12-04_19:23:11.52852 sql-trace:17:23:11.528459 create table if not exists “systems” (“name” varchar(64) not null primary key, “value” varchar(1024)) ; [] (625.548µs)
2015-12-04_19:23:11.52971 sql-trace:17:23:11.529614 create table if not exists “incomingwebhooks” (“id” varchar(26) not null primary key, “createat” bigint, “updateat” bigint, “deleteat” bigint, “userid” varchar(26), “channelid” varchar(26), “teamid” varchar(26)) ; [] (1.069575ms)
2015-12-04_19:23:11.53104 sql-trace:17:23:11.530991 create table if not exists “outgoingwebhooks” (“id” varchar(26) not null primary key, “token” varchar(26), “createat” bigint, “updateat” bigint, “deleteat” bigint, “creatorid” varchar(26), “channelid” varchar(26), “teamid” varchar(26), “triggerwords” varchar(1024), “callbackurls” varchar(1024)) ; [] (1.261652ms)
2015-12-04_19:23:11.53202 sql-trace:17:23:11.531999 create table if not exists “preferences” (“userid” varchar(26) not null, “category” varchar(32) not null, “name” varchar(32) not null, “value” varchar(128), primary key (“userid”, “category”, “name”)) ; [] (939.945µs)
2015-12-04_19:23:11.53354 sql-trace:17:23:11.533514 SELECT COUNT(0)
2015-12-04_19:23:11.53356 FROM pg_attribute
2015-12-04_19:23:11.53356 WHERE attrelid = $1::regclass
2015-12-04_19:23:11.53357 AND attname = $2
2015-12-04_19:23:11.53357 AND NOT attisdropped [1:“teams” 2:“allowvalet”] (1.487768ms)
2015-12-04_19:23:11.53631 sql-trace:17:23:11.536282 SELECT COUNT(0)
2015-12-04_19:23:11.53633 FROM pg_attribute
2015-12-04_19:23:11.53633 WHERE attrelid = $1::regclass
2015-12-04_19:23:11.53634 AND attname = $2
2015-12-04_19:23:11.53634 AND NOT attisdropped [1:“teams” 2:“inviteid”] (2.732702ms)
2015-12-04_19:23:11.53729 sql-trace:17:23:11.537249 SELECT COUNT(0)
2015-12-04_19:23:11.53731 FROM pg_attribute
2015-12-04_19:23:11.53731 WHERE attrelid = $1::regclass
2015-12-04_19:23:11.53732 AND attname = $2
2015-12-04_19:23:11.53732 AND NOT attisdropped [1:“teams” 2:“allowopeninvite”] (940.006µs)
2015-12-04_19:23:11.53909 sql-trace:17:23:11.539069 SELECT COUNT(0)
2015-12-04_19:23:11.53914 FROM pg_attribute
2015-12-04_19:23:11.53914 WHERE attrelid = $1::regclass
2015-12-04_19:23:11.53915 AND attname = $2
2015-12-04_19:23:11.53915 AND NOT attisdropped [1:“teams” 2:“allowteamlisting”] (1.795961ms)
2015-12-04_19:23:11.54057 sql-trace:17:23:11.540546 SELECT COUNT(0)
2015-12-04_19:23:11.54059 FROM pg_attribute
2015-12-04_19:23:11.54060 WHERE attrelid = $1::regclass
2015-12-04_19:23:11.54060 AND attname = $2
2015-12-04_19:23:11.54061 AND NOT attisdropped [1:“channelmembers” 2:“notifyprops”] (1.448837ms)
2015-12-04_19:23:11.54153 sql-trace:17:23:11.541503 SELECT COUNT(0)
2015-12-04_19:23:11.54154 FROM pg_attribute
2015-12-04_19:23:11.54155 WHERE attrelid = $1::regclass
2015-12-04_19:23:11.54155 AND attname = $2
2015-12-04_19:23:11.54157 AND NOT attisdropped [1:“channels” 2:“description”] (933.268µs)
2015-12-04_19:23:11.54340 sql-trace:17:23:11.543370 SELECT COUNT(0)
2015-12-04_19:23:11.54342 FROM pg_attribute
2015-12-04_19:23:11.54343 WHERE attrelid = $1::regclass
2015-12-04_19:23:11.54344 AND attname = $2
2015-12-04_19:23:11.54344 AND NOT attisdropped [1:“channels” 2:“purpose”] (1.838333ms)
2015-12-04_19:23:11.55172 sql-trace:17:23:11.551637 SELECT data_type FROM INFORMATION_SCHEMA.COLUMNS where table_name = $1 AND column_name = $2 [1:“Posts” 2:“Props”] (8.169587ms)
2015-12-04_19:23:11.56683 sql-trace:17:23:11.566777 ALTER TABLE Posts ALTER COLUMN Props TYPE text [] (15.051974ms)
2015-12-04_19:23:11.56805 sql-trace:17:23:11.567999 SELECT COUNT(0)
2015-12-04_19:23:11.56808 FROM pg_attribute
2015-12-04_19:23:11.56809 WHERE attrelid = $1::regclass
2015-12-04_19:23:11.56810 AND attname = $2
2015-12-04_19:23:11.56811 AND NOT attisdropped [1:“users” 2:“themeprops”] (1.182443ms)
2015-12-04_19:23:11.57058 sql-trace:17:23:11.570528 SELECT $1::regclass [1:“idx_teams_name”] (2.477552ms)
2015-12-04_19:23:11.57283 sql-trace:17:23:11.572783 SELECT $1::regclass [1:“idx_teams_invite_id”] (2.201852ms)
2015-12-04_19:23:11.57400 sql-trace:17:23:11.573954 SELECT $1::regclass [1:“idx_channels_team_id”] (1.125055ms)
2015-12-04_19:23:11.57575 sql-trace:17:23:11.575705 SELECT $1::regclass [1:“idx_channels_name”] (1.705999ms)
2015-12-04_19:23:11.57675 sql-trace:17:23:11.576663 SELECT $1::regclass [1:“idx_channelmembers_channel_id”] (911.691µs)
2015-12-04_19:23:11.57846 sql-trace:17:23:11.578396 SELECT $1::regclass [1:“idx_channelmembers_user_id”] (1.686063ms)
2015-12-04_19:23:11.57988 sql-trace:17:23:11.579836 SELECT $1::regclass [1:“idx_posts_update_at”] (1.387152ms)
2015-12-04_19:23:11.58099 sql-trace:17:23:11.580947 SELECT $1::regclass [1:“idx_posts_create_at”] (1.030067ms)
2015-12-04_19:23:11.58224 sql-trace:17:23:11.582193 SELECT $1::regclass [1:“idx_posts_channel_id”] (1.199245ms)
2015-12-04_19:23:11.58372 sql-trace:17:23:11.583668 SELECT $1::regclass [1:“idx_posts_root_id”] (1.42926ms)
2015-12-04_19:23:11.58529 sql-trace:17:23:11.585243 SELECT $1::regclass [1:“idx_posts_message_txt”] (1.528946ms)
2015-12-04_19:23:11.58706 sql-trace:17:23:11.587007 SELECT $1::regclass [1:“idx_posts_hashtags_txt”] (1.719568ms)
2015-12-04_19:23:11.58982 sql-trace:17:23:11.589757 SELECT $1::regclass [1:“idx_users_team_id”] (2.704072ms)
2015-12-04_19:23:11.59202 sql-trace:17:23:11.591975 CREATE INDEX idx_users_team_id ON Users (TeamId) [] (2.174727ms)

`
Does anyone have a solution to this problem?

Can you verify the index idx_users_team_id exists? You can created it manually with something like CREATE INDEX idx_users_team_id ON Users (TeamId)

Has this been solved?

We have almost the identical issue since upgrading from 8.0 to gitlab 8.4. Mattermost seems to be upgraded implicitely from 0.7 to 1.4.

The column that is not created in my system is called “inviteid” instead of teamid:

2016-01-25_13:50:29.02835 panic: Failed to create index pq: column "inviteid" does not exist

Postgres constantly keeps logging this:

...
2016-01-25_15:05:09.78374 STATEMENT:  CREATE INDEX idx_teams_invite_id ON Teams (InviteId)
2016-01-25_15:05:10.80553 ERROR:  relation "idx_teams_invite_id" does not exist
2016-01-25_15:05:10.80555 STATEMENT:  SELECT $1::regclass
2016-01-25_15:05:10.80584 ERROR:  column "inviteid" does not exist
2016-01-25_15:05:10.80585 STATEMENT:  CREATE INDEX idx_teams_invite_id ON Teams (InviteId)
...

@coreyhule : How could I check if index exists. I installed Gitlab with omnibus, so everything runs out of the box. Could I query mattermost db in command line somehow?

I suspect you’re missing more than just idx_teams_invite_id if you’ve skipped a bunch of mattermost versions. Anything before 1.0 didn’t support upgrade. After 1.0 you need to install every version to make sure it gets upgraded correctly. You can see https://github.com/mattermost/platform/blob/master/doc/install/Upgrade-Guide.md#gitlab-mattermost-upgrade-troubleshooting for more info.

Seems like all trouble with missing columns and Bad Gateway responses from Nginx had to do with skipped versions. Here ist how we got it solved:

  • download all major mattermost releases that you have skipped. I had upgraded from Gitlab 8.0 to 8.4 which had skipped the following mattermost versions:
    – 1.0.0
    – 1.1.0
    – 1.2.1
    – 1.3.0
  • download each version and extract them in to seperate folders
  • stop mattermost (the correct command on my system was gitlab-ctl stop mattermost)
  • go to /opt/gitlab/embedded/bin/mattermost and make a backup copy from the file mattermost
  • replace it with the file platform from the bin folder of mattermost 1.0.0
  • rename the platform file to mattermost
  • start mattermost by gitlab-ctl start mattermost

The GUI won’t work, but you may trace in your logs that mattermost started and updated your database. Repeat the replacing of the mattermost file with the platform binary of each mattermost version. Last but not least, place your latest version of mattermost in gitlab.