Segmentation fault while triggering webhook

Troubleshooting
1

Summary

Server consistently shut down with SIGSEGV when triggering a specific webook.

Steps to reproduce

Mattermost 5.4.0

Expected behavior

Successful webhook post, or error in log

Observed behavior

Several attempts to post messages via the webhook failed because the destination hostname could not be resolved. Those errors were visible in the log.
After that, there was nothing else in the log immediately prior to the event. Presumably another trigger word was encountered, and then we saw:

Dec 14 03:23:57 <server> start-mattermost.sh[114318]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 14 03:23:57 <server> start-mattermost.sh[114318]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0xeb8acd]
Dec 14 03:23:57 <server> start-mattermost.sh[114318]: goroutine 50655814 [running]:
Dec 14 03:23:57 <server> start-mattermost.sh[114318]: github.com/mattermost/mattermost-server/app.(*App).TriggerWebhook.func1.1()
Dec 14 03:23:57 <server> start-mattermost.sh[114318]: /home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/app/webhook.go:108 +0xfd
Dec 14 03:23:57 <server> start-mattermost.sh[114318]: github.com/mattermost/mattermost-server/app.(*App).Go.func1(0xc00f3e18b0, 0xc000100580)
Dec 14 03:23:57 <server> start-mattermost.sh[114318]: /home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/app/app.go:505 +0x27
Dec 14 03:23:57 <server> start-mattermost.sh[114318]: created by github.com/mattermost/mattermost-server/app.(*App).Go
Dec 14 03:23:57 <server> start-mattermost.sh[114318]: /home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/app/app.go:504 +0x52

This crash was repeated a further two times.

2

Hi @gubbins,

Would you be open to sharing more details on your config settings?

3

Yes, what do you need?

4

Hi @gubbins,

Thank you - could you share your config.json file as well as your installation environment?

5

Installation is on CentOS 7.

Sharing the entire config file is not very practical - is there a relevant section or setting?

From the code (webhook.go:108) it looks like maybe http.NewRequest returned null, which is interesting given that we were having difficulty resolving the destination hostname. I hoped it might be reproducible just from that.

6

I tested an incoming webhook, a slash command and the outgoing webhook in the 5.4 version and I was not able to reproduce the issue.
is it possible you share with me the configuration you did (like if a slash command or other) and the message that you are trying to post

thanks

1 Like
7

@cpanato It was an outgoing webhook, posting JSON to a URL like http://nn.nn.nn.nn:5000/status.

I do not believe the webhook settings themselves are the problem. I believe the problem is that in some circumstances http.NewRequest may return null if the destination URL is not resolvable.

8

https://github.com/mattermost/mattermost-server/blob/master/app/webhook.go: in TriggerWebhook:

				req, _ := http.NewRequest("POST", url, body)
				req.Header.Set("Content-Type", contentType)
				req.Header.Set("Accept", "application/json")

Clearly we are not expecting req to ever be nil here, but I believe the stack trace I posted at the top shows that it was indeed nil.

https://github.com/golang/go/blob/2012227b01020eb505cf1dbe719b1fa74ed8c5f4/src/net/http/request.go: in NewRequest it looks like nil is returned if the URL cannot be parsed.

I tried an experiment on our dev server - I set up an outgoing webhook with URL http://arealhostname:5000/whoops#this%&breaks;=no (with a valid hostname) and reproduced the crash.

I think the server should just be checking for the nil response here. Sorry for the lack of a PR - I would submit one if I had more time and was actually a golang developer!

9

@cpanato Do you know if this seems to be a bug and I should create a Jira ticket?

10

@amy.blais yes please create a jira ticket

11

thanks for reporting this, we will check that internally now :slight_smile:

12

Opened a ticket here: https://mattermost.atlassian.net/browse/MM-13659.