Can't connect to S3 - error checking if bucket exists

Troubleshooting
1

Summary

I’ve set up my Mattermost instance to upload files to S3, but I get an error:

Connection unsuccessful: Error checking if bucket exists.

Steps to reproduce

I’m using Mattermost 5.21.0 via Bitnami on AWS EC2.

Expected behavior

The ListBucket check should pass and users should be able to upload images.

Observed behavior

Here’s what I’ve tried:

  • created a private s3 bucket at mattermost-media (name redacted) in eu-central-1
  • created an IAM user with the following policy:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::mattermost-media/*"
        }
    ]
}
  • filled the bucket name, region id, access key id, and secret access key to the Mattermost system console

The ListBucket check doesn’t pass, here is the error I get in the logs: (I enabled the AWS degub option)

{
  "level": "error",
  "ts": 1586443405.2713053,
  "caller": "mlog/log.go:175",
  "msg": "Error checking if bucket exists.",
  "path": "/api/v4/file/s3_test",
  "request_id": "myrjkoh4kfn1md9phj3ijbjjhr",
  "ip_addr": "87.123.193.33",
  "user_id": "rp8196yejigwty5yfhby5e4uzw",
  "method": "POST",
  "err_where": "TestFileConnection",
  "http_code": 500,
  "err_details": "Access Denied."
}

When I try to upload an image, I get:

{
  "level": "error",
  "ts": 1586441803.4454176,
  "caller": "app/file.go:805",
  "msg": "Unable to upload",
  "path": "20200409/teams/noteam/channels/fic5dh7843d9ib34cmtczsbrwy/users/rp8196yejigwty5yfhby5e4uzw/43fo9njwh7n73mpbw5e3bemwde/photo.jpg",
  "error": "WriteFile: Encountered an error writing to S3, Access Denied"
}

Are there some known caveats when setting up S3 that I might have missed? For example, I didn’t edit the Amazon S3 Endpoint field, should I have?

Otherwise, how can I debug this further?

Thank you in advance for your support!
Robin

Update

About half an hour after writing this post, I was able to upload an image, and it’s in the S3 bucket as expected. However, the listBicket check is still failing. It seems to be a little unstable - any help is welcome!

2

@paulrothrock Would support team be familiar with this issue?

3

@robinmetral, the minio client under the covers makes a HEAD request against the root to determine if the bucket exists and can be listed. I’m not certain, but I wonder if the /* in your Resource configuration is actually restricting HEAD access on just /? From the configuration examples on Deprecation of the MinIO gateway, only a Resource of arn:aws:s3:::mattermost-media /should/ be required.

Would love to hear if this works!

1 Like
4

I tested things out with a few more policies:

First attempt

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::mattermost-media"
            ]
        }
    ]
}

As you mentioned @jesse, this should work, I actually initially started with it. However, when I switch now, the media that were uploaded in the past are not found :thinking: Is it possible this changes the object URLs?

Second attempt

 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:*:*:accesspoint/*",
                "arn:aws:s3:::*/*",
                "arn:aws:s3:*:*:job/*",
                "arn:aws:s3:::mattermost-media"
            ]
        }
    ]
}

This policy was adjusted based on AWS recommendations given in the IAM console. I’m not exactly sure why, but this passes the listBuket check, and the old media are still being fetched properly.

I think I’ll keep my current policy (see the question) for now, because media uploads work as expected even though the listBucket check fails. If anything comes up, I might switch to the “Second attempt” policy with obscure resources :slight_smile: