We're Hiring!

Mattermost, Inc.

After upgrade to 5.37 seeing "http: panic serving" errors

For feature requests, please see: Contributing Feature Proposals - Mattermost.

For troubleshooting questions, please post in the following format:

Summary
Upgraded from 5.33.2 to 5.37. After the upgrade, I started seeing multiple error http/server.go:3093 http: panic serving 127.0.0.1:53512: runtime error: invalid memory address or nil pointer dereference errors in the mattermost.log

Steps to reproduce
Open the System Console

Expected behavior
no errors logged

Observed behavior
I upgraded from 5.33.2 to 5.37. When I upgraded, I went from 5.33.2 → 5.33.5 - > 5.34.5 → 5.35.4 → 5.36.1 → 5.37, each upgrade one at a time.

Currently running the server on ubuntu 16.04.7, mysql 5.7.33.

During the upgrades, I didn’t notice any failures or errors.

After the upgrade, every time I go into the system console, I see the http: panic crash errors like this:

2021-07-17T13:59:06.870-0500	error	http/server.go:3093	http: panic serving 127.0.0.1:53512: runtime error: invalid memory address or nil pointer dereference
goroutine 9663 [running]:
net/http.(*conn).serve.func1(0xc0050b3900)
	net/http/server.go:1801 +0x147
panic(0x2521aa0, 0x3f2d9d0)
	runtime/panic.go:975 +0x47a
github.com/getsentry/sentry-go/http.(*Handler).recoverWithSentry(0xc000cdc9e0, 0xc004d38b10, 0xc0054af300)
	github.com/getsentry/sentry-go@v0.11.0/http/sentryhttp.go:117 +0x14f
panic(0x2521aa0, 0x3f2d9d0)
	runtime/panic.go:969 +0x1b9
github.com/mattermost/mattermost-server/v5/api4.getPrevTrialLicense(0xc004204180, 0x2cf16e0, 0xc004204140, 0xc0054af500)
	github.com/mattermost/mattermost-server/v5/api4/license.go:278 +0x63
github.com/mattermost/mattermost-server/v5/web.Handler.ServeHTTP(0x2d679a0, 0xc001dda740, 0x29d65a8, 0x3ceacd8, 0x13, 0x100000001, 0x0, 0x0, 0x2cf4060, 0xc0040b7960, ...)
	github.com/mattermost/mattermost-server/v5/web/handlers.go:298 +0x1db2
github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0x2cf4060, 0xc0040b7960, 0xc0054af500)
	github.com/NYTimes/gziphandler@v1.1.1/gzip.go:343 +0x2f0
net/http.HandlerFunc.ServeHTTP(0xc0013f1d40, 0x2cf4060, 0xc0040b7960, 0xc0054af500)
	net/http/server.go:2042 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc00024a300, 0x2cf4060, 0xc0040b7960, 0xc0054af300)
	github.com/gorilla/mux@v1.8.0/mux.go:210 +0xd3
github.com/getsentry/sentry-go/http.(*Handler).handle.func1(0x2cf4060, 0xc0040b7960, 0xc0054af200)
	github.com/getsentry/sentry-go@v0.11.0/http/sentryhttp.go:103 +0x402
net/http.HandlerFunc.ServeHTTP(0xc002d6e5c0, 0x2cf4060, 0xc0040b7960, 0xc0054af200)
	net/http/server.go:2042 +0x44
github.com/mattermost/mattermost-server/v5/app.(*RateLimiter).RateLimitHandler.func1(0x2cf4060, 0xc0040b7960, 0xc0054af200)
	github.com/mattermost/mattermost-server/v5/app/ratelimit.go:107 +0xc8
net/http.HandlerFunc.ServeHTTP(0xc002d6e600, 0x2cf4060, 0xc0040b7960, 0xc0054af200)
	net/http/server.go:2042 +0x44
net/http.serverHandler.ServeHTTP(0xc005a4a0e0, 0x2cf4060, 0xc0040b7960, 0xc0054af200)
	net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0050b3900, 0x2cfdf20, 0xc002b00840)
	net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
	net/http/server.go:2969 +0x36c	{"source": "httpserver"}

Is there a way to figure out what is causing these? Could it be an incompatibility with the server running ubuntu 16.04?

Best regards,
Christopher

This is listed as a known in the changelog. The logs are polluted from the errors you posted, but there are are no functional effects and Mattermost should work properly. A dev is looking into a fix via [MM-37108] Fix possible nil dereference in team edition by streamer45 · Pull Request #17933 · mattermost/mattermost-server · GitHub & [MM-37108] Only make request if server is enterprise ready by streamer45 · Pull Request #8406 · mattermost/mattermost-webapp · GitHub PRs.

1 Like

Hi Amy,

Doh! Somehow I had completely missed that in the change log. Thanks!

Best regards,
Christopher