Is this by design?

Is it by design that any attachment from any channel can be accessed by an unauthenticated party knowing the URL?
It seems that, at the bare minimum posted content from a private channel shouldn’t be available to anyone who has been given a link.

Have you enabled public attachments and made that specific attachments public?

System Console > Public Links