Outgoing Webhook sometimes fails to POST

Summary

I get the following error sometimes when typing outgoing webhook keywords into mattermost:

Event POST failed, err=Post http://mywebendpoint.com/somepath/somefile.json: dial tcp: i/o timeout

Steps to reproduce

Type in the webhook keyword. Press enter to submit.
This tends to happen if I haven’t typed the webhook keyword in the last minute or so. Perhaps it’s related to websockets not playing nice? Webserver is apache and I don’t think it’s set up to handle websockets well, just plain http requests.

Expected behavior

Webhook should post a request to my server endpoint any time a keyword is triggered.

Observed behavior

Sometimes typing in a keyword doesn’t post a request to my server. Typically when it’s the first time someone has used the hook in the last minute or so.

Hi @maddawgx9,

Thanks for your feedback,

Perhaps this documentation will help?

If you still have issues after going through the webhook set-up documentation, please let us know which version of Mattermost you have installed and a little more about your installation environment so we can help troubleshoot?

I reviewed that documentation thoroughly while setting up the webhook. I have yet to resolve this issue. We are currently running the mattermost server version 3.10.0 on one server, and the web endpoiont it is attempting to reach is returning json, is on a separate (physical) server, and is being hosted up by an apache web server. The web endpoint seems to be working fine as it responds to curl requests properly.

The interesting part is the mattermost integration works most of the time, the only time it fails to post is upon the first outgoing hook triggering. If you retype the trigger word after the initial failure, it works just fine.

Thanks @maddawgx9,

Interesting… I’ll post your issue to the team and see whether they are aware of this issue and/or have some troubleshooting ideas.

You could try upgrading to the latest version (4.1) of Mattermost which might help solve your issue.

Hi @maddawgx9,

How quickly does your webhook usually respond to the server? I’m not sure what the exact timeout is for those requests, but it might be taking too long to respond.

Also, is it possible to confirm if your webhook is receiving anything from the server in those cases? Like does it have its own logs that might explain more?

The endpoint responds pretty quick to my curl testing. Certainly not long enough to warrant any reasonable timeout. It has it’s own logs and when it fails, it doesn’t even hit my server endpoint, it’s like mattermost tries to post and can’t reach the endpoint server at all on the first try, but then if I try immediately afterward it’s completely fine.

I will try upgrading to the latest version and report back.

Thanks @maddawgx9,

Let us know how it goes and if upgrading resolves the issue for you

Just upgraded the server this morning, still getting the same results.

Also, I ran a check on the response time. The endpoint server responds to curl requests made from the mattermost server properly with around 78ms response time.

Hi @maddawgx9,

That response time definitely sounds fine, so that shouldn’t be the issue. You said the web server and Mattermost are on separate machines, so is there anything between them that could be causing the connection to be interrupted, like a firewall or proxy? If so, they might have additional logs that could explain why the connection appears to be dropped

No proxy or firewall that I know of. If that were the problem, I would expect the curl requests would fail in a similar fashion.

No ideas why a mattermost post/get request would act different from a curl request?

Sorry, I’m not sure what the issue could be. The only other thing I could think is that the endpoint appears to be returning a static file instead of something dynamic, but I can’t see how that would cause it to time out. If it was an https request, it might be a certificate issue that could be fixed by changing the EnableInsecureOutgoingConnections setting, but it looks like you’re using plain http and that also wouldn’t make sense if it was only a periodic issue.

1 Like

You’re the best! I completely forgot that our endpoint redirects http to https. Switching EnableInsecureOutgoingConnections to true seems to have fixed the issue.

Awesome. I was just throwing ideas out there, so I’m glad to hear that fixed it.