Mattermost, Inc.

Mattermost EBS Docker restart = wipe?

I got some weird logs and I am a total docker newb.

Essentially I think in the middle of the night SOMETHING ordered by docker images to kill, pull new images and restart.
Did this wipe all data? I can not connect in any way?

TO i have to set it up new now?

time=“2017-11-09T01:36:04.867090444Z” level=error msg="Error closing logger: invalid argument"
time=“2017-11-09T01:36:05.017211714Z” level=error msg="Error closing logger: invalid argument"
time=“2017-11-09T01:36:34.944860918Z” level=info msg="Container d04e7a4f07bac67b257a30c3457bac61aab17e24a81bad310993e4765d0f9052 failed to exit within 30 seconds of signal 15 - using the force"
time=“2017-11-09T01:36:34.977124807Z” level=error msg="Error closing logger: invalid argument"
time=“2017-11-09T04:36:35.125368311Z” level=error msg="Handler for GET /v1.17/containers/d04e7a4f07bac67b257a30c3457bac61aab17e24a81bad310993e4765d0f9052/json returned error: No such container: d04e7a4f07bac67b257a30c3457bac61aab17e24a81bad310993e4765d0f9052"
time=“2017-11-09T04:36:35.151044017Z” level=error msg="Handler for GET /v1.17/containers/63244ccdeb756414ad5ae1e908fa9c41b916f8e5c883d70717fa59bc54590524/json returned error: No such container: 63244ccdeb756414ad5ae1e908fa9c41b916f8e5c883d70717fa59bc54590524"
time=“2017-11-09T04:36:35.178356785Z” level=error msg=“Handler for GET /v1.17/containers/8a443558b7cfaf55aa6a645427e8fd0900798d9075b60c018861fdcfdc940d0e/json returned error: No such container: 8a443558b7cfaf55aa6a645427e8fd0900798d9075b60c018861fdcfdc940d0e”

AND DOCKER

2017-11-09T01:36:04.835212438Z container oom 63244ccdeb756414ad5ae1e908fa9c41b916f8e5c883d70717fa59bc54590524 (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=app, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/2fcd82fa-0ce5-4ae3-a4de-a5dd73b59b67, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, image=mattermost/mattermost-prod-app:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-app-feceb6a797a9e1e76e00)
2017-11-09T01:36:04.865975159Z container die 63244ccdeb756414ad5ae1e908fa9c41b916f8e5c883d70717fa59bc54590524 (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=app, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/2fcd82fa-0ce5-4ae3-a4de-a5dd73b59b67, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, exitCode=137, image=mattermost/mattermost-prod-app:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-app-feceb6a797a9e1e76e00)
2017-11-09T01:36:04.925816336Z network disconnect 1c2b8f8ca2c5ea17fc4ed7bcd3ff74b0a6822e521eab45dd8a85a1e57a294311 (container=63244ccdeb756414ad5ae1e908fa9c41b916f8e5c883d70717fa59bc54590524, name=bridge, type=bridge)
2017-11-09T01:36:04.943714908Z container kill 8a443558b7cfaf55aa6a645427e8fd0900798d9075b60c018861fdcfdc940d0e (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=db, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/2fcd82fa-0ce5-4ae3-a4de-a5dd73b59b67, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, image=mattermost/mattermost-prod-db:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-db-80aaf9c3c4e1a9d23d00, signal=15)
2017-11-09T01:36:04.944652013Z container kill d04e7a4f07bac67b257a30c3457bac61aab17e24a81bad310993e4765d0f9052 (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=web, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/2fcd82fa-0ce5-4ae3-a4de-a5dd73b59b67, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, image=mattermost/mattermost-prod-web:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-web-f8aba5edc3a286d8ef01, signal=15)
2017-11-09T01:36:05.016357319Z container die 8a443558b7cfaf55aa6a645427e8fd0900798d9075b60c018861fdcfdc940d0e (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=db, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/2fcd82fa-0ce5-4ae3-a4de-a5dd73b59b67, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, exitCode=0, image=mattermost/mattermost-prod-db:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-db-80aaf9c3c4e1a9d23d00)
2017-11-09T01:36:05.071591068Z network disconnect 1c2b8f8ca2c5ea17fc4ed7bcd3ff74b0a6822e521eab45dd8a85a1e57a294311 (container=8a443558b7cfaf55aa6a645427e8fd0900798d9075b60c018861fdcfdc940d0e, name=bridge, type=bridge)
2017-11-09T01:36:05.106661591Z container stop 8a443558b7cfaf55aa6a645427e8fd0900798d9075b60c018861fdcfdc940d0e (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=db, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/2fcd82fa-0ce5-4ae3-a4de-a5dd73b59b67, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, image=mattermost/mattermost-prod-db:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-db-80aaf9c3c4e1a9d23d00)
2017-11-09T01:36:34.945255272Z container kill d04e7a4f07bac67b257a30c3457bac61aab17e24a81bad310993e4765d0f9052 (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=web, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/2fcd82fa-0ce5-4ae3-a4de-a5dd73b59b67, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, image=mattermost/mattermost-prod-web:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-web-f8aba5edc3a286d8ef01, signal=9)
2017-11-09T01:36:34.976370556Z container die d04e7a4f07bac67b257a30c3457bac61aab17e24a81bad310993e4765d0f9052 (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=web, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/2fcd82fa-0ce5-4ae3-a4de-a5dd73b59b67, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, exitCode=137, image=mattermost/mattermost-prod-web:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-web-f8aba5edc3a286d8ef01)
2017-11-09T01:36:35.060616438Z network disconnect 1c2b8f8ca2c5ea17fc4ed7bcd3ff74b0a6822e521eab45dd8a85a1e57a294311 (container=d04e7a4f07bac67b257a30c3457bac61aab17e24a81bad310993e4765d0f9052, name=bridge, type=bridge)
2017-11-09T01:36:35.086695719Z container stop d04e7a4f07bac67b257a30c3457bac61aab17e24a81bad310993e4765d0f9052 (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=web, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/2fcd82fa-0ce5-4ae3-a4de-a5dd73b59b67, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, image=mattermost/mattermost-prod-web:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-web-f8aba5edc3a286d8ef01)
2017-11-09T01:37:21.290354422Z image pull mattermost/mattermost-prod-db:latest (name=mattermost/mattermost-prod-db)
2017-11-09T01:37:21.514610523Z container create c3d598083c1fc17fbb5350ae953f5626eeed50b8c03b3b189ac97e75b994cd48 (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=db, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/202a5049-d798-4caa-bbe5-6ea348640924, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, image=mattermost/mattermost-prod-db:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-db-a48a8685c9a5b5da2f00)
2017-11-09T01:37:21.619307624Z network connect 1c2b8f8ca2c5ea17fc4ed7bcd3ff74b0a6822e521eab45dd8a85a1e57a294311 (container=c3d598083c1fc17fbb5350ae953f5626eeed50b8c03b3b189ac97e75b994cd48, name=bridge, type=bridge)
2017-11-09T01:37:21.777113921Z container start c3d598083c1fc17fbb5350ae953f5626eeed50b8c03b3b189ac97e75b994cd48 (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=db, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/202a5049-d798-4caa-bbe5-6ea348640924, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, image=mattermost/mattermost-prod-db:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-db-a48a8685c9a5b5da2f00)
2017-11-09T01:37:36.349864687Z image pull mattermost/mattermost-prod-app:latest (name=mattermost/mattermost-prod-app)
2017-11-09T01:37:36.531183006Z container create 6f7950de693175515a0715fb7be360093131fb63f1f4ca20efaec1816e1aa12c (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=app, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/202a5049-d798-4caa-bbe5-6ea348640924, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, image=mattermost/mattermost-prod-app:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-app-94b581aeade1c4c4fc01)
2017-11-09T01:37:36.616736206Z network connect 1c2b8f8ca2c5ea17fc4ed7bcd3ff74b0a6822e521eab45dd8a85a1e57a294311 (container=6f7950de693175515a0715fb7be360093131fb63f1f4ca20efaec1816e1aa12c, name=bridge, type=bridge)
2017-11-09T01:37:36.774615924Z container start 6f7950de693175515a0715fb7be360093131fb63f1f4ca20efaec1816e1aa12c (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=app, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/202a5049-d798-4caa-bbe5-6ea348640924, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, image=mattermost/mattermost-prod-app:latest, name=ecs-awseb-vmatter-env-5kmccibvya-1-app-94b581aeade1c4c4fc01)
2017-11-09T01:37:44.541797676Z image pull mattermost/mattermost-prod-web:latest (maintainer=NGINX Docker Maintainers docker-maint@nginx.com, name=mattermost/mattermost-prod-web)
2017-11-09T01:37:44.739401584Z container create ddc2b0f3b1ca142f06b30d5cd686fa709f38e61ea9317244239dc9839b2ff43d (com.amazonaws.ecs.cluster=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.container-name=web, com.amazonaws.ecs.task-arn=arn:aws:ecs:eu-west-1:241520883193:task/202a5049-d798-4caa-bbe5-6ea348640924, com.amazonaws.ecs.task-definition-family=awseb-vmatter-env-5kmccibvya, com.amazonaws.ecs.task-definition-version=1, image=mattermost/mattermost-prod-web:latest, maintainer=NGINX Docker Maintainers docker-maint@nginx.com, name=ecs-awseb-vmatter-env-5kmccibvya-1-web-b8d59efa97da8bc85000)
2017-11-09T01:37:44.892305410Z network connect 1c2b8f8ca2c5ea17fc4ed7bcd3ff74b0a6822e521eab45dd8a85a1e57a294311 (container=ddc2b0f3b1ca142f06b30d5cd686fa709f38e61ea9317244239dc9839b2ff43d, name=bridge, type=bridge)

Basically no but this sounds like an issue between AWS EBS and Docker.

May I know how did you setup your Mattermost. Have you followed this documentation https://docs.mattermost.com/install/docker-ebs.html?

Were you able to successfully run Mattermost instance?

yes we have run this successfully for 3 months.

then a spike in traffic. basically redownlaoding the containers I think and set it up again without me doing anything.

I feel everything was wiped.

Not really sure what happened to your instance but the container oom means out of memory. I’d suggest to sort out the problem with AWS/Docker first and let us know your findings.