Mattermost, Inc.

Mattermost-push-proxy. Test notification using curl works, but DMs are not triggering notifs

Hello.
Having trouble with setting up mattermost-push-proxy service.
I’m receiving notifications as expected when I test my notification service using curl (according to https://developers.mattermost.com/contribute/mobile/push-notifications/service/#test-the-mattermost-push-notification-service).
However, notifications are not getting triggered on DMs and mentions.

Services versions:

mattermost-push-proxy: 5.21
mattermost-server: 5.23.0
mattermost-mobile: 1.31.2
mattermost-webapp: 5.23.0

Here is my systemd unit file. Service running on Ubuntu 18.04.

[Unit]

Description=Mattermost Push Proxy

[Service]

StartLimitInterval=200

StartLimitBurst=5

WorkingDirectory=/home/ubuntu/mattermost-push-proxy

User=mattermost-push-proxy

ExecStart=/home/ubuntu/mattermost-push-proxy/bin/mattermost-push-proxy | logger

[Install]

WantedBy=multi-user.target

Below are the logs for push-proxy service

Jun 11 11:19:04 rocketchat mattermost-push-proxy[45621]: 2020/06/11 11:19:04 INFO Loading /home/ubuntu/mattermost-push-proxy/config/mattermost-push-proxy.json

Jun 11 11:19:04 rocketchat mattermost-push-proxy[45621]: 2020/06/11 11:19:04 INFO Push proxy server is initializing…

Jun 11 11:19:04 rocketchat mattermost-push-proxy[45621]: 2020/06/11 11:19:04 INFO Initializing apple notification server for type=apple

Jun 11 11:19:04 rocketchat mattermost-push-proxy[45621]: 2020/06/11 11:19:04 ERROR Apple push notifications not configured. Missing ApplePushCertPrivate. for type=apple

Jun 11 11:19:04 rocketchat mattermost-push-proxy[45621]: 2020/06/11 11:19:04 INFO Initializing apple notification server for type=apple_rn

Jun 11 11:19:04 rocketchat mattermost-push-proxy[45621]: 2020/06/11 11:19:04 ERROR Apple push notifications not configured. Missing ApplePushCertPrivate. for type=apple_rn

Jun 11 11:19:04 rocketchat mattermost-push-proxy[45621]: 2020/06/11 11:19:04 INFO Initializing Android notification server for type=android

Jun 11 11:19:04 rocketchat mattermost-push-proxy[45621]: 2020/06/11 11:19:04 INFO Initializing Android notification server for type=android_rn

Jun 11 11:19:04 rocketchat mattermost-push-proxy[45621]: 2020/06/11 11:19:04 INFO Server is listening on :8066

Jun 11 11:19:44 rocketchat mattermost-push-proxy[45621]: 2020/06/11 11:19:44 INFO Sending android push notification for device=android_rn and type=message

The last log is from testing the service using curl (according to https://developers.mattermost.com/contribute/mobile/push-notifications/service/#test-the-mattermost-push-notification-service). I am also receiving the test notification on my android device.
However, I’m not receiving any notifs from DMs or mentions. I’ve enabled notifications in account settings for all activity.

Below is the log from mattermost.log

{“level”:“error”,“ts”:1591874402.2690194,“caller”:“mlog/log.go:175”,“msg”:“An error occurred sending the receipt delivery to the push notification service.”,“path”:"/api/v4/notifications/ack",“request_id”:“ukup1qu1qfyjbb9gkgx7x7jfrc”,“ip_addr”:“59.99.200.114”,“user_id”:“1hnqc5ghoibnbfcgos4y6mibuc”,“method”:“POST”,“err_where”:“pushNotificationAck”,“http_code”:500,“err_details”:“Post “http://rocketchat.eastus.cloudapp.azure.com:8066/api/v1/ack”: dial tcp 13.68.190.193:8066: i/o timeout”}

{“level”:“error”,“ts”:1591874406.7174678,“caller”:“mlog/log.go:175”,“msg”:“An error occurred sending the receipt delivery to the push notification service.”,“path”:"/api/v4/notifications/ack",“request_id”:“yweh8s7tc3dybmzapn5ax8rnah”,“ip_addr”:“59.99.200.114”,“user_id”:“1hnqc5ghoibnbfcgos4y6mibuc”,“method”:“POST”,“err_where”:“pushNotificationAck”,“http_code”:500,“err_details”:“Post “http://rocketchat.eastus.cloudapp.azure.com:8066/api/v1/ack”: dial tcp 13.68.190.193:8066: i/o timeout”}

{“level”:“error”,“ts”:1591874412.0932093,“caller”:“mlog/log.go:175”,“msg”:“An error occurred sending the receipt delivery to the push notification service.”,“path”:"/api/v4/notifications/ack",“request_id”:“csjz3i7hfpn9uqc5435fjaphhe”,“ip_addr”:“59.99.200.114”,“user_id”:“1hnqc5ghoibnbfcgos4y6mibuc”,“method”:“POST”,“err_where”:“pushNotificationAck”,“http_code”:500,“err_details”:“Post “http://rocketchat.eastus.cloudapp.azure.com:8066/api/v1/ack”: dial tcp 13.68.190.193:8066: i/o timeout”}

{“level”:“error”,“ts”:1591874418.3994908,“caller”:“mlog/log.go:175”,“msg”:“An error occurred sending the receipt delivery to the push notification service.”,“path”:"/api/v4/notifications/ack",“request_id”:“c47xbe45dbr53c85kdjp969xqh”,“ip_addr”:“59.99.200.114”,“user_id”:“1hnqc5ghoibnbfcgos4y6mibuc”,“method”:“POST”,“err_where”:“pushNotificationAck”,“http_code”:500,“err_details”:“Post “http://rocketchat.eastus.cloudapp.azure.com:8066/api/v1/ack”: dial tcp 13.68.190.193:8066: i/o timeout”}

{“level”:“error”,“ts”:1591874440.5887835,“caller”:“mlog/log.go:175”,“msg”:“An error occurred sending the receipt delivery to the push notification service.”,“path”:"/api/v4/notifications/ack",“request_id”:“uzs8nbboy3gsdkc464gwwg1jxc”,“ip_addr”:“59.99.200.114”,“user_id”:“1hnqc5ghoibnbfcgos4y6mibuc”,“method”:“POST”,“err_where”:“pushNotificationAck”,“http_code”:500,“err_details”:“Post “http://rocketchat.eastus.cloudapp.azure.com:8066/api/v1/ack”: dial tcp 13.68.190.193:8066: i/o timeout”}

Below is the log from notifications.log

{“level”:“info”,“ts”:1591874724.7103631,“caller”:“app/notification_push.go:310”,“msg”:“Notification will be sent”,“logSource”:“notifications”,“ackId”:“ktcwypig7jg7td9wf5du7na8cr”,“type”:“message”,“userId”:“a7uwmmwnyiy5dm3j4qjynne3bh”,“postId”:“ffwzk3b5r3y6me1u7iqeuf8jfh”,“status”:“Prepared to send”}

{“level”:“error”,“ts”:1591874727.7107913,“caller”:“app/notification_push.go:106”,“msg”:“Notification error”,“logSource”:“notifications”,“ackId”:“ktcwypig7jg7td9wf5du7na8cr”,“type”:“message”,“userId”:“a7uwmmwnyiy5dm3j4qjynne3bh”,“postId”:“ffwzk3b5r3y6me1u7iqeuf8jfh”,“channelId”:“f7opni4ir3rsdgu85qk85fxbza”,“deviceId”:“elVi1rRAUL4:APA91bGzQxXBoVk04UcW2n6MjK_8WFyRykN8T7w837LmnDqZTW288AC-0m3iWHrJPuxqB1kBar3v81bfgtG98XH_5FMVpLqEgcbX5c6urLZVfZKMHbDR64R7L1_K7njqcNRZRAQ4W-yw”,“status”:“Post “http://rocketchat.eastus.cloudapp.azure.com:8066/api/v1/send_push”: dial tcp 13.68.190.193:8066: i/o timeout”}

{“level”:“info”,“ts”:1591874727.710909,“caller”:“app/notification_push.go:310”,“msg”:“Notification will be sent”,“logSource”:“notifications”,“ackId”:“m9ozfnzu1fyrdpc86ke7ozz71h”,“type”:“message”,“userId”:“a7uwmmwnyiy5dm3j4qjynne3bh”,“postId”:“ffwzk3b5r3y6me1u7iqeuf8jfh”,“status”:“Prepared to send”}

{“level”:“error”,“ts”:1591874730.7113843,“caller”:“app/notification_push.go:106”,“msg”:“Notification error”,“logSource”:“notifications”,“ackId”:“m9ozfnzu1fyrdpc86ke7ozz71h”,“type”:“message”,“userId”:“a7uwmmwnyiy5dm3j4qjynne3bh”,“postId”:“ffwzk3b5r3y6me1u7iqeuf8jfh”,“channelId”:“f7opni4ir3rsdgu85qk85fxbza”,“deviceId”:“fLOIW6BFmQU:APA91bG0GnXLh9myublX_AEU8U2YPi8eJL1IpZpsFfezGYmGxY4e0nLx1Q0GE2vWuqBTv7oBSv0r5V-U1xFw6CKb-fTo1CZGBeFaj3N188-PS7oUrnpXaHG66d2IbU2tb-OfwLgzHoUN”,“status”:“Post “http://rocketchat.eastus.cloudapp.azure.com:8066/api/v1/send_push”: dial tcp 13.68.190.193:8066: i/o timeout”}

{“level”:“info”,“ts”:1591874730.711515,“caller”:“app/notification_push.go:310”,“msg”:“Notification will be sent”,“logSource”:“notifications”,“ackId”:“chbyj41n43fhfcy19rgofzkm5w”,“type”:“message”,“userId”:“a7uwmmwnyiy5dm3j4qjynne3bh”,“postId”:“ffwzk3b5r3y6me1u7iqeuf8jfh”,“status”:“Prepared to send”}

{“level”:“error”,“ts”:1591874733.711911,“caller”:“app/notification_push.go:106”,“msg”:“Notification error”,“logSource”:“notifications”,“ackId”:“chbyj41n43fhfcy19rgofzkm5w”,“type”:“message”,“userId”:“a7uwmmwnyiy5dm3j4qjynne3bh”,“postId”:“ffwzk3b5r3y6me1u7iqeuf8jfh”,“channelId”:“f7opni4ir3rsdgu85qk85fxbza”,“deviceId”:“dEH7GScjxPk:APA91bEL33rTZRZ2cMdw8nSN10-ZL3s_o3Cf4ZhSnxfscB4kQyAVdFy_NFsp9uEn_qPNRHgFDyBWbphwOXSVJ9yCVu2S9hkD2mzFx5kZAp3qnZjpwRAtJ-rGaL3V2s3KmnvANdlvJ7CQ”,“status”:“Post “http://rocketchat.eastus.cloudapp.azure.com:8066/api/v1/send_push”: dial tcp 13.68.190.193:8066: i/o timeout”}

{“level”:“info”,“ts”:1591874733.7120166,“caller”:“app/notification_push.go:310”,“msg”:“Notification will be sent”,“logSource”:“notifications”,“ackId”:“j5cjy5m7winnzkiixizanhzc6r”,“type”:“message”,“userId”:“a7uwmmwnyiy5dm3j4qjynne3bh”,“postId”:“ffwzk3b5r3y6me1u7iqeuf8jfh”,“status”:“Prepared to send”}

{“level”:“error”,“ts”:1591874736.712317,“caller”:“app/notification_push.go:106”,“msg”:“Notification error”,“logSource”:“notifications”,“ackId”:“j5cjy5m7winnzkiixizanhzc6r”,“type”:“message”,“userId”:“a7uwmmwnyiy5dm3j4qjynne3bh”,“postId”:“ffwzk3b5r3y6me1u7iqeuf8jfh”,“channelId”:“f7opni4ir3rsdgu85qk85fxbza”,“deviceId”:“dEM42smJkbI:APA91bEW7jUg83Osl3Irn_e7c2Q8w2HadW_gAeG6wpRVu0FyvnnZDoKbXrNL3nrtXkfFT6OUPzHCavzRD1iNMhLYifxDVBB9Te2n3e4B_GzTyRj9AAEA9X60Hmz2-EWOVEO215qFRXGK”,“status”:“Post “http://rocketchat.eastus.cloudapp.azure.com:8066/api/v1/send_push”: dial tcp 13.68.190.193:8066: i/o timeout”}

{“level”:“info”,“ts”:1591874736.7124076,“caller”:“app/notification_push.go:310”,“msg”:“Notification will be sent”,“logSource”:“notifications”,“ackId”:“f31u3zh9hbdiiedy8tmzo61nxw”,“type”:“message”,“userId”:“a7uwmmwnyiy5dm3j4qjynne3bh”,“postId”:“ffwzk3b5r3y6me1u7iqeuf8jfh”,“status”:“Prepared to send”}

{“level”:“error”,“ts”:1591874739.7128098,“caller”:“app/notification_push.go:106”,“msg”:“Notification error”,“logSource”:“notifications”,“ackId”:“f31u3zh9hbdiiedy8tmzo61nxw”,“type”:“message”,“userId”:“a7uwmmwnyiy5dm3j4qjynne3bh”,“postId”:“ffwzk3b5r3y6me1u7iqeuf8jfh”,“channelId”:“f7opni4ir3rsdgu85qk85fxbza”,“deviceId”:“cLVZa0AUDrc:APA91bEqqsjK9K5c_HyJnBwjDnyzTleXuq7TX6A-zW7ZVBB-TCJqwQ4upc8D2Nwrl7zUidZSkHdAJzdenYfgqb_jN3R8i48Yt-hLeK0frjgtoD6CGeIXFFu_TnVpAKxmCbJTOB6VZVFe”,“status”:“Post “http://rocketchat.eastus.cloudapp.azure.com:8066/api/v1/send_push”: dial tcp 13.68.190.193:8066: i/o timeout”}

Below is the Push Notification Service configuration in my mattermost instance.

Please kindly help.