Permission error creating new dm and uploading files (rhel + posgresql)

Troubleshooting
1

Migrated from one server to another while updating from 3.3.x to 4.x and have been getting a bunch of errors in the logs. Some of the issues are not being able to create new direct message channels via the UI (a workaround is typing /msg @user text which will bring up a new direct message). Also some users failing to upload files. I have gone through and checked permissions which seem mostly correct (match original server). It is running on RHEL 6.6 and Postgresql 9.4. Any help appreciated.

[2017/08/31 18:24:53 PDT] [EROR] /api/v4/channels/direct:Permissions code=403 rid=j6zbhzapeirednn5rbuiq53tew uid=h1n5kx9q9iy7tnrikxds1omhxr ip=2.2.2.2 You do not have the appropriate permissions [details: userId=h1n5kx9q9iy7tnrikxds1omhxr, permission=create_direct_channel]
[2017/08/31 18:24:53 PDT] [EROR] /api/v4/channels/direct:Permissions code=403 rid=pupnc9uc9pb3zbxdgbfbrbkare uid=h1n5kx9q9iy7tnrikxds1omhxr ip=2.2.2.2 You do not have the appropriate permissions [details: userId=h1n5kx9q9iy7tnrikxds1omhxr, permission=create_direct_channel]
[2017/08/31 18:27:13 PDT] [EROR] /api/v4/channels/dhueuf6wujft5gxpefu73pjfwa/members/me:SqlChannelStore.GetMember code=500 rid=rtawokqmwfbcz8awfgp4bdtcmr uid=jbbeh9mc4id5fyz5iac5icc4wh ip=3.3.3.3 We couldn't get the channel member [details: channel_id=dhueuf6wujft5gxpefu73pjfwauser_id=jbbeh9mc4id5fyz5iac5icc4wh,gorp: multiple rows returned for: SELECT * FROM ChannelMembers WHERE ChannelId = :ChannelId AND UserId = :UserId - [map[ChannelId:dhueuf6wujft5gxpefu73pjfwa UserId:jbbeh9mc4id5fyz5iac5icc4wh]]]
[2017/08/31 18:27:13 PDT] [EROR] /api/v4/channels/dhueuf6wujft5gxpefu73pjfwa/members/me:SqlChannelStore.GetMember code=500 rid=s11nh9thofnjxjo3fsh657nysa uid=jbbeh9mc4id5fyz5iac5icc4wh ip=3.3.3.3 We couldn't get the channel member [details: channel_id=dhueuf6wujft5gxpefu73pjfwauser_id=jbbeh9mc4id5fyz5iac5icc4wh,gorp: multiple rows returned for: SELECT * FROM ChannelMembers WHERE ChannelId = :ChannelId AND UserId = :UserId - [map[ChannelId:dhueuf6wujft5gxpefu73pjfwa UserId:jbbeh9mc4id5fyz5iac5icc4wh]]]
[2017/08/31 18:31:05 PDT] [EROR] /api/v4/channels/x9p71x81jifwtmp8ccamxy55sr/members/me:SqlChannelStore.GetMember code=500 rid=pjk41m4c7fb7inwb18pqxcnmic uid=x7kgxoo3h7bsxcrx9jzbn5tb3o ip=1.1.1.1 We couldn't get the channel member [details: channel_id=x9p71x81jifwtmp8ccamxy55sruser_id=x7kgxoo3h7bsxcrx9jzbn5tb3o,gorp: multiple rows returned for: SELECT * FROM ChannelMembers WHERE ChannelId = :ChannelId AND UserId = :UserId - [map[ChannelId:x9p71x81jifwtmp8ccamxy55sr UserId:x7kgxoo3h7bsxcrx9jzbn5tb3o]]]
[2017/08/31 18:31:07 PDT] [EROR] /api/v4/channels/y97hbb43tbnppndfg6nsfce6ya/members/me:SqlChannelStore.GetMember code=500 rid=arxhzpjd6jnsdn967fh9ohfmja uid=x7kgxoo3h7bsxcrx9jzbn5tb3o ip=1.1.1.1 We couldn't get the channel member [details: channel_id=y97hbb43tbnppndfg6nsfce6yauser_id=x7kgxoo3h7bsxcrx9jzbn5tb3o,gorp: multiple rows returned for: SELECT * FROM ChannelMembers WHERE ChannelId = :ChannelId AND UserId = :UserId - [map[ChannelId:y97hbb43tbnppndfg6nsfce6ya UserId:x7kgxoo3h7bsxcrx9jzbn5tb3o]]]
[2017/08/31 18:31:08 PDT] [EROR] /api/v4/channels/y3jaz38reprb7qj3wq89umu3sc/members/me:SqlChannelStore.GetMember code=500 rid=oyqb91uebbgd38khkuq7x3765y uid=x7kgxoo3h7bsxcrx9jzbn5tb3o ip=1.1.1.1 We couldn't get the channel member [details: channel_id=y3jaz38reprb7qj3wq89umu3scuser_id=x7kgxoo3h7bsxcrx9jzbn5tb3o,gorp: multiple rows returned for: SELECT * FROM ChannelMembers WHERE ChannelId = :ChannelId AND UserId = :UserId - [map[ChannelId:y3jaz38reprb7qj3wq89umu3sc UserId:x7kgxoo3h7bsxcrx9jzbn5tb3o]]]
[2017/08/31 18:31:13 PDT] [EROR] /api/v4/channels/dhueuf6wujft5gxpefu73pjfwa/members/me:SqlChannelStore.GetMember code=500 rid=nmap39cdfp8zunwjqksjapna6o uid=x7kgxoo3h7bsxcrx9jzbn5tb3o ip=1.1.1.1 We couldn't get the channel member [details: channel_id=dhueuf6wujft5gxpefu73pjfwauser_id=x7kgxoo3h7bsxcrx9jzbn5tb3o,gorp: multiple rows returned for: SELECT * FROM ChannelMembers WHERE ChannelId = :ChannelId AND UserId = :UserId - [map[ChannelId:dhueuf6wujft5gxpefu73pjfwa UserId:x7kgxoo3h7bsxcrx9jzbn5tb3o]]]
[2017/08/31 18:38:31 PDT] [EROR] /api/v4/channels/direct:Permissions code=403 rid=remj1mpd5br5j8r4f9gjgj375r uid=h1n5kx9q9iy7tnrikxds1omhxr ip=1.1.1.1 You do not have the appropriate permissions [details: userId=h1n5kx9q9iy7tnrikxds1omhxr, permission=create_direct_channel]
[2017/08/31 18:38:32 PDT] [EROR] /api/v4/channels/direct:Permissions code=403 rid=o6i48434ht87fxg4i84kex4tfe uid=h1n5kx9q9iy7tnrikxds1omhxr ip=2.2.2.2 You do not have the appropriate permissions [details: userId=h1n5kx9q9iy7tnrikxds1omhxr, permission=create_direct_channel]
[2017/08/31 18:38:33 PDT] [EROR] /api/v4/channels/direct:Permissions code=403 rid=8g5gjojg1pf1dd7w6ntq4u9r5a uid=h1n5kx9q9iy7tnrikxds1omhxr ip=2.2.2.2 You do not have the appropriate permissions [details: userId=h1n5kx9q9iy7tnrikxds1omhxr, permission=create_direct_channel]
[2017/08/31 18:38:34 PDT] [EROR] /api/v4/channels/direct:Permissions code=403 rid=n9pgw7b7p38kumcamzn6hd966a uid=h1n5kx9q9iy7tnrikxds1omhxr ip=1.1.1.1 You do not have the appropriate permissions [details: userId=h1n5kx9q9iy7tnrikxds1omhxr, permission=create_direct_channel]
[2017/08/31 18:41:31 PDT] [EROR] /api/v4/users/search: code=401 rid=wnufh5nwt3fofr5x14gg9ekgoc uid= ip=1.1.1.1 Invalid or expired session, please login again. [details: UserRequired]
[2017/08/31 18:45:22 PDT] [EROR] /api/v4/channels/dhueuf6wujft5gxpefu73pjfwa/members/me:SqlChannelStore.GetMember code=500 rid=he96ptff7brbbebrxwixpy9xxh uid=x7kgxoo3h7bsxcrx9jzbn5tb3o ip=1.1.1.1 We couldn't get the channel member [details: channel_id=dhueuf6wujft5gxpefu73pjfwauser_id=x7kgxoo3h7bsxcrx9jzbn5tb3o,gorp: multiple rows returned for: SELECT * FROM ChannelMembers WHERE ChannelId = :ChannelId AND UserId = :UserId - [map[ChannelId:dhueuf6wujft5gxpefu73pjfwa UserId:x7kgxoo3h7bsxcrx9jzbn5tb3o]]]
[2017/08/31 18:45:35 PDT] [EROR] /api/v4/channels/dhueuf6wujft5gxpefu73pjfwa/members/me:SqlChannelStore.GetMember code=500 rid=6d5ojifw5ibd5xgzrr3848k6wc uid=x7kgxoo3h7bsxcrx9jzbn5tb3o ip=1.1.1.1 We couldn't get the channel member [details: channel_id=dhueuf6wujft5gxpefu73pjfwauser_id=x7kgxoo3h7bsxcrx9jzbn5tb3o,gorp: multiple rows returned for: SELECT * FROM ChannelMembers WHERE ChannelId = :ChannelId AND UserId = :UserId - [map[ChannelId:dhueuf6wujft5gxpefu73pjfwa UserId:x7kgxoo3h7bsxcrx9jzbn5tb3o]]]
2

Seems to be primarily an issue with api v4 calls

3

The creating new direct message seems to be caused by the cache/cookie. Solved by clearing browser cache or deleting %USERPROFILE%\AppData\Roaming\Mattermost\Cookies for Mattermost application. Still the error for read and writing files.

[2017/09/01 16:02:11 PDT] [EROR] /api/v4/channels/c7uic7a9n7fxfkoizcw4j3k7fc/members/me:SqlChannelStore.GetMember code=500 rid=nk8o1i3mbtnixbcrqqyrjkimwy uid=t638nroywpdf8rww7pzfz5p73c ip=xxx.xxx.xxx.xxx We couldn’t get the channel member [details: channel_id=c7uic7a9n7fxfkoizcw4j3k7fcuser_id=t638nroywpdf8rww7pzfz5p73c,gorp: multiple rows returned for: SELECT * FROM ChannelMembers WHERE ChannelId = :ChannelId AND UserId = :UserId - [map[ChannelId:c7uic7a9n7fxfkoizcw4j3k7fc UserId:t638nroywpdf8rww7pzfz5p73c]]]
[2017/09/01 16:02:11 PDT] [EROR] /api/v4/posts/kcq4iib1b3fw7yjkmd4mbumnmw/files/info:Permissions code=403 rid=tkgbyy98qt8hdqtnhs8j7fn17o uid=t638nroywpdf8rww7pzfz5p73c ip=xxx.xxx.xxx.xxx You do not have the appropriate permissions [details: userId=t638nroywpdf8rww7pzfz5p73c, permission=read_channel]

4

To replicate the issue, please check the following and let us know. thanks!

  1. When you moved from one server to the other, were you able to test/check if the same version worked well (like the new server is working well with 3.3.x)?
  2. Could you specifically state which version 4 you’ve upgraded to (e.g. 4.1.0)?
  3. For permissions errors like when creating direct channel, could you confirm whether the user involved has system_user role?
  4. For reading or writing files, are you using local file system? Does mattermost has ownership over mattermost installation folder including the local storage directory (e.g. ./data/)?
  5. Could you compare your config.json against the default config.json of version 4?
5
  1. I have not tested an older 3.0 version on the new server
  2. 4.1.0 using this guide https://docs.mattermost.com/administration/legacy-upgrade.html
  3. where can i check the system_user role?
  4. yes, it’s a local filesystem with mattermost ownership
  5. I overwrote the 4.1.0 config.json with the one from 3.3.0

Thanks for the response.

6
  1. If you have access to your database, you’ll find it in Roles column of Users table. Also, please check Roles column at ChannelMembers (should have channel_user and/or channel_admin) and TeamMembers (team_user and/or team_admin) tables. All users should have role and not empty.
  2. Could you try to change a setting in System Console, then revert the change you’ve made? That’s just to restore missing item(s) in config setting and have it set to default. Just to make sure that all items in are in your config.json file. Do the refresh, then verify how it works after.
7

Users table looks correct. There are a lot of rows in ChannelMembers table that are missing ‘channel_user’. Team user looks correct as well. Did a refresh on the config.json but seems to be the same so I’m guessing it’s because of the roles in ChannelMembers table.

8

Yeah, channel member should have atleast channel_user role in order to have permission to upload/download files, read, etc.

9

I added channel_user where ever there was an empty role but still the same permission errors:

[EROR] /api/v4/posts/qtm83kq9afndpykcsztbk7z79a/files/info:Permissions code=403 rid=cykf9rznxtr9xedfsi6ykswyxy uid=gfz3dim967bguehhaywphbqouc ip=X.X.X.X You do not have the appropriate permissions [details: userId=gfz3dim967bguehhaywphbqouc, permission=read_channel]

I restarted/reloaded sql and restarted mattermost. Is there any other reload that needs to be done?

10

You may purge all caches at System Console >> Configuration, then click Purge All Caches.

11

Unfortunately that didn’t seem to do anything. Sending a file to a new channel (a channel created after the update) even if the users were created before the update works fine. I’m trying to see if there is anything missing in the channels table.