We're Hiring!

Mattermost, Inc.

Upgrade to 5.33.0 caused major 'bad file descriptor' issue

Summary

Upgrade from 5.32.1 to 5.33.0 brings on strange behaviors in the environment, such as double posts, posts being attributed to the wrong person, and high resource utilization

Steps to reproduce

Running v5.33.0 seems to have created issues. I reverted back to v5.32.1 and the environment has stabilized

Expected behavior

Describe your issue in detail

Observed behavior

There are 1000’s of these messages being logged per minute.

The only commit i can find in the mattermost-server Github that i think may be a factor is:

Same library, similar issue. How does this tie into Mattermost?

Would you be open to upgrading to v5.33.1 to see if it fixes the issue?

1 Like

Sure thing. This must have JUST been released this morning. I did the upgrade at 5AM today, and 5.33.0 was the latest.

2 Likes

I had the same issue and someone popped 5.33.1 which fixed it.

Hey there @efreeman - apologies for the inconvenience. Yes 5.33.1 fixes the issue you are seeing.

1 Like

I just deployed 5.33.1, and the bad file descriptor errors are gone. I am seeing a lot of these errors in the logs, but i don’t see anything out of ordinary from the point-of-view of the desktop application.

2021-03-23T06:58:10.352Z error http/h2_bundle.go:4215 http2: server connection error from 141.157.218.32:64168: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:58:10.352Z error http/h2_bundle.go:4215 http2: server connection error from 141.157.218.32:64168: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:58:16.144Z error http/h2_bundle.go:4215 http2: server connection error from 108.41.225.235:53722: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:58:47.667Z error http/h2_bundle.go:4215 http2: server connection error from 74.64.160.204:54363: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:58:48.195Z error http/h2_bundle.go:4215 http2: server connection error from 71.202.126.110:51378: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:58:53.220Z error http/h2_bundle.go:4215 http2: server connection error from 69.248.58.194:57399: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:58:53.220Z error http/h2_bundle.go:4215 http2: server connection error from 69.248.58.194:57399: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:59:06.760Z error http/h2_bundle.go:4215 http2: server connection error from 119.18.2.179:10739: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:59:11.394Z error http/h2_bundle.go:4215 http2: server connection error from 71.190.230.144:55136: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:59:18.434Z error http/h2_bundle.go:4215 http2: server connection error from 100.2.223.183:50846: connection error: PROTOCOL_ERROR {“source”: “httpserver”}
2021-03-23T06:59:25.251Z error http/h2_bundle.go:4215 http2: server connection error from 141.149.48.58:64020: connection error: PROTOCOL_ERROR {“source”: “httpserver”}

I see. Were you not getting these errors earlier? We have seen these errors in the past, and it is due to an issue with the Go standard library which was just recently fixed (i.e. not yet in a released Go version). But it shouldn’t have to do anything with websockets.

OK. Thank you all so much for the assistance! Our Mattermost server has stabilized now, and we’re all set on 5.33.1!

2 Likes