feature request: Configure "PROXY Protocol" for Dovecot + Postfix #3866
Comments
cfis
added
kind/new feature
polarathene
changed the title
polarathene
added
service/dovecot
service/postfix
area/tests
area/features
kind/improvement
|
I don't have bandwidth available to provide much feedback on this right now (EDIT: Ok.. semi-tackled it). But glancing over at the config you referenced there should be easier ways to support that.
Looking at our PROXY Protocol docs page, where does it suggest anything about these extra ports like you've configured? ( Issues wise, there's only your recent helm chart update request mentioning them too: #3825 (and the associated helm project README port section) You document many of the ports we support with All I have to go by is this snippet in our docs:
These are internal services, but why is that an issue? I'd need to reproduce this to verify but it sounds odd to me.
Neither of those have any relevance to port References:
While that HAProxy blog post does put port 25 on the reverse proxy, and adjusts Postfix to use a different port internally, this isn't applicable for containers where each container has a separate IP with it's own set of ports. Perhaps some misunderstandings have carried over from that? I see no indication for such a requirement. The docs for Dovecot settings is a bit different though. They seem more strict implying that the Proxy Protocol header must be received and that it must be from a client in the configured trusted networks for haproxy connections. We do have issues reporting internal services that don't route through the reverse proxy are incompatible with Postfix as well:
Another public port (optional) is I came across this mailcow commit that applied the same config ports for Postfix / Dovecot, but it lacks a PR reference for any other context. If the ports are not used internally by anything however, then it should be fine to just toggle proxy protocol support on the existing service ports. Nothing should really change then in usage, provided all traffic external of that container is going through the reverse proxy service to reach DMS. One last thing, you've defined a set of trusted networks for proxy are those accurate? (If I understand correctly, it's a configurable setting?) For Docker at least, the networks it supports are from a pool of large subnets:
Thus if you had two containers in the same network, rather than a This sort of trust btw is a bit risky when IPv6 is enabled, at least for Docker where the daemon has Oh and your doc links should also prefer to use the version ( |
|
Regarding actual support being implemented. If the concerns above can be addressed first for if the additional ports are actually needed / beneficial, that'd be appreciated. Implementing the support shouldn't be too troublesome, but I'd like some test coverage which is where some friction will likely be introduced for the contributor. Presently we don't run tests with If other maintainers aren't fussed about test coverage, but see it as a worthwhile contribution beyond guidance offered in docs being updated, I guess I'd be ok with a reproduction I can use locally verify the contribution proposed is necessary. |
|
The helm chart was already using port 10993,
Port 25 - yes - it should be added.
For the text you pasted, yes, I'm not sure if that is true or not - thanks for looking into that. But here is the real use case and why this is important.
I did test this a fair bit on my cluster:
The trusted networks are easily configurable (just override in a custom values.yaml). And in fact, I do override them because I need to trust the IPs of the pods inside my cluster). The default values were already part of the Helm chart and are the three IPV4 non-routable ranges. Meaning they likely are (must be?) computers on your network and therefore should be trustworthy. So I thought it was a reasonable default and kept it. I think it is worth adding PROXY support to DMS if there are cases, like in Kubernetes, where you need it to figure out the client ip address (seems like it - moby/moby#15086). If not, then it can stay in the Helm chart. If it stay in the Helm chart, would love to know if there are better ways to configure it, especially the postfix setup. I used |
Yes, I don't maintain the helm chart repo as I have no real k8s experience, likewise with the docs. I have very little familiarity with ProxyProtocol too. I have however pointed out very clearly that the justification in our docs page is very odd/wrong regarding port The other link you provided is for a non-container setup (only one service can bind a specific port per IP, thus separate ports is necessary). They put the regular ports to connect to only via the reverse proxy configured (HAProxy) and only provide the alternative ports in Postfix and Dovecot. You're proposing to have both available from these services on the DMS container though. The LinuxBabe article makes the real ports available via the "frontend" network, which then go through the ProxyProtocol "backend" ports. This doesn't help justify your use-case for other local services connecting to the real ports on DMS directly though, since that's not what is configured there either.
Port 25 has two roles, inbound and outbound connections.
If you don't proxy inbound port 25 for the same reasons you're proxying the others, restrictions are bypassed. You permit receiving mail without DKIM/DMARC/SPF checks for example which is quite bad. The issue with an open relay that I was raising was with trusted networks (such as Postfix NOTE: Don't forget the mentioned concern with ManageSieve Dovecot service also needing to be configured for proxy.
You can have the same kind of deployment with Docker Compose, but how you connect to the DMS container can change by configuring DNS and networks. Take Roundcube as a separate container in your cluster. How would you deploy this container when all users connecting through it in their browsers delegates the auth process through the Roundcube container to the DMS container? This is another known scenario where the single IP source of the Roundcube container is problematic, should a user fail to login enough times that Fail2Ban triggers and bans that IP, all users are now banned via that service. Roundcube does support ProxyProtocol, but this would only work when going through the compatible port (either directly to DMS or through the reverse proxy with ProxyProtocol support on both inbound (from RoundCube) + outbound (to DMS) traffic). When that sort of problem described with Roundcube is not a concern, the only reason you'd need the non-proxied ports is for convenience of a direct connection between the two services without the proxy involved?
I stated in my previous comment that the ProxyProtocol feature in both Postfix and Dovecot enforce that requirement AFAIK, so I'm aware 👍
It's a very wide trust. DMS defaults to not trusting these because of security risks that introduces (especially with some networking gotchas/bugs I know of with Docker). I'd kind of suggest the same approach if you care about security, then this is something the user should opt-in for trust. You'll find that reverse proxies like Caddy and Traefik explicitly need opt-in for trusted networks, they don't just assume the private ranges are safe to trust. I can have two VMs with their own private internal networks, but the two VMs can reach each other via a shared network. While they normally couldn't access the private internal networks of the other VM, such as those managed by Docker, Docker can configure the network via a kernel tunable which unintentionally allows the other VM to route traffic through the shared network to the private network. This is enabled by default IIRC, but not applicable to connections from public IPs, still it's a vulnerability concern for some deployments like at a company where an employee with a system on the network could compromise. I verified it in private cloud networks too, so all it'd take is one server compromised by a malicious user to access the containers of another that were assumed private. With Docker at least, my point was that the subnet a container belongs in is notably narrower, not sure how that applies to k8s networking. I might trust my PC at home, but I don't trust the devices of others on the same network.
See my comment in the issue you referenced: moby/moby#15086 (comment) If the reverse proxy is a container as well, it wouldn't matter since the real client IP is already lost in the scenario many users would experience from IPv6 / userland-proxy. Docker specific AFAIK. May have been some other cases where the networking rules introduced other issues there and something like ProxyProtocol may have helped sure. I'm not saying it isn't worth adding the feature you're requesting. I'm just questioning the approach, especially with duplicating ports vs sending all ingress through the proxy instead of direct to DMS?
I don't have the time to put much thought into this beyond the feedback given here. If you were just toggling proxy support instead of adding new ports, you'd have it handled with a few commands:
For your current approach with postfix, I believe there is a Dovecot config, you can find the line |
|
Agreed on adding ports 25 and 4190 (ManageSieve). Retrieving (yes, not receiving) / sending email in the cluster...Roundcube is a good example and is in fact another application I have running in my cluster. Roundcube talks directly to the docker-mailserver service (and thus pod) on the relevant ports (465/587 or 993). There is no use of the proxy protocol because this is internal cluster traffic. The proxy protocol only applies to traffic incoming to the cluster. In this case, a user would be using port 443 from their local browser to talk to the roundcube service (no proxy there) and roundcube in turn talks to docker-mailserver to retrieve/send email (no proxy there either). Proxy support comes into play if you are using an external client, such as Thunderbird, and it talks to docker mailserver. It will send traffic on ports 465/587/993 to the cluster, and those ports are proxied and directed to ports 10465/10587/10993 which have been configured to understand the proxy protocol.
Not only is that a big convenience, it is essential since most apps know nothing about the proxy protocol. Gitlab? NextCloud? Photo sharing apps (that want to send notifications)? Etc?
Because that would mean traffic in the cluster would have to go outside the cluster, do a hair-pin turn, and then come back in the cluster via the proxy. I'm not even sure it would work once you have more than one host (but I'm far from a K8 networking expert).
Every K8 pod on any host can directly to any other pod on any other host. That is all internal cluster traffic. That is the main goal of K8 networking. So that means any app that wants to retrieve/send email just talks directly to the docker mailserver service no matter where in the cluster it might be deployed. Having two sets of ports enables both exterior cluster clients and interior clients. Having just one set of ports breaks one of those two use cases. |
Ok, but if someone were to deploy Roundcube + DMS, how does this avoid the issue with the IP of all remote users authenticating through Roundcube being the same? Isn't ProxyProtocol required?
My understanding was:
I'm only aware of the issue here being when Fail2Ban is enabled within DMS, since Roundcube delegates the user auth to the users DMS account. However with the source IP always appearing the same from Roundcube regardless of user authenticating, this is a compatibility issue. Doesn't the ProxyProtocol fix that? For other services it's not a concern, since they should all be submitting mail through a single account for that service specifically, not tied to any individual user of the service.
Yes I understand this. It should be just as viable to only use 465/587/993 though. Postfix/Dovecot only expecting ProxyProtocol and any connection should be through the reverse proxy that speaks ProxyProtocol. Why is that not viable?
I'm sorry, why do they need to know ProxyProtocol to connect through the reverse proxy? Take the NextCloud SMTP config docs for example.
For any external client, like via ThunderBird for example this is already configured the same way to connect to DMS via DNS resolving the IP and connecting to that port. Why is this different in your internal services? It should all resolve to the reverse proxy container IP + port. Doing so will then route seamlessly to DMS with ProxyProtocol for everyone, and only the reverse proxy and DMS Postfix/Dovecot services need to care about that. In other circumstances like with Roundcube, you can do a little more configuration to ensure ProxyProtocol is used when connecting to the reverse proxy (Traefik makes this optional at least, so no conflicting compatibility concerns there). To me these are separate concerns:
The latter is relevant for an alternative
I don't have the k8s expertise to chime in the networking situation there. With Docker AFAIK the traffic within the same subnet will resolve directly to the reverse proxy regardless of internal vs external DNS. It has hairpin rules in iptables established related to this IIRC. But in the sense of local DNS resolution, I don't see why this would be any different when we're just talking about resolving to a different container IP? Perhaps you're relying on hostnames already configured for you implicitly in In Docker I can just add a host directly: $ docker run --rm -it --add-host 'reverse-proxy:10.20.30.40' --hostname some-container alpine cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
10.20.30.40 reverse-proxy
172.17.0.3 some-containerHow you go about assigning IPs is up to you, I have a specific network for a group of containers to belong in that can access each other, if needed I can choose that subnet and explicitly assign which IP belongs to a container. That's all just the same to me like ENV config is. Surely k8s has something similar to that. So really, why can't you configure to send a different hostname that maps to the reverse-proxy which recognizes something like # Quick example by spinning up a container:
# This hostname added has no relevance anywhere else, DMS is configured as mail.origin.test, IP is assigned dynamically
$ docker run --rm -it --add-host dms-internal.example.test:172.18.0.4 --network my-dms-network alpine:edge
# Add swaks to send a test mail:
$ apk add swaks --repository=https://dl-cdn.alpinelinux.org/alpine/edge/testing
# Send that test mail:
$ swaks --server dms-internal.example.test:587 --from alpine@wherever.test --to john.doe@origin.test --auth-user jane.doe@origin.test --auth-pass secret
# Success!Simple as that. Should be no different if that host/server configured pointed to a reverse proxy container, so long as it has an entry configured to route for that?
I'm not convinced. All I can see is a convenience in routing traffic directly, which may complicate troubleshooting when that difference introduces some subtle bug.... and then we the maintainers get burdened by issue reports. We've already established config mistakes in the helm chart and our proxy protocol docs page, and how these have been relied upon rather than better understanding the why (or that reasoning being invalid), which doesn't bring much confidence that maintainers will be able to troubleshoot issues related to the feature (just see the past issue history on the subject). I'm open towards a PR(s) that implements the needed support, but I don't see the need to complicate our maintenance with a feature that duplicates ports redundantly for convenience.
Reference from alternative projects:
Do you have some other references where this need for duplicate ports is supported elsewhere (with context of k8s deployments / containers)? Do you have examples for anything else that isn't mail related? I'm sorry about the friction, just trying to avoid the XY problem. DMS has taken contributions in the past for features that weren't given much review consideration, and we ended up with messes like the current relay host feature and LDAP support. I just want to get a clear understanding of what is needed and the value that actually offers to the bulk of our users. |
Traffic to roundcube goes is via HTTP(S) via a kubernetes ingress. That uses a different mechanism, called
Because clients inside the cluster are not using the Proxy Protocol. And their traffic does not go through the Proxy because they are communicating with each other entirely inside the cluster.
Which requires traffic to leave the cluster to then come back into the cluster. Kubernetes networking is different than Docker. Kubernetes has a layer of indirection (services), its own DNS server in which it tracks services, and then pods. Pods get IP addresses from the same subnet, you can't assign specific ip addresses to specific pods. The whole point of this setup is a flat address space to make it easy for pods to talk to each other through services, no matter what host they are running on. Maybe this helps - https://kubernetes.io/docs/concepts/services-networking/ In my view, forcing traffic outside of the cluster is working against the way Kubernetes is designed. I assume you could probably make it work but, at least to me, seems like a clearly a worse solution than having two sets of mail ports. |
|
Another thought - I have been mostly assuming you are deploying docker-mailserver to a bare metal K8 cluster that is running on a home server or EC2 instance. But if you are running in a managed K8 cluster like EKS, then traffic into the cluster is coming in via an external load balancer controlled by the hosting provider (ie, AWS, Azure, GCP). Now, this is not my area of expertise, so I may be totally wrong on this. But my reading of this: Is that if you have an app (roundcube) that makes a network call outside the cluster that goes to a load balancer that routes back to the same node (say where docker-mailserver happens to be) then it will timeout if you want to preserve the client ip address. So you would have to disable that, which then of course makes using the Proxy protocol pointless. Which is my main worry - sending traffic outside the cluster to come back in seems likely to introduce unexpected networking issues that will differ depending on your specific Kubernetes setup and provider. |
Interior clients are actually something that is very common, and something that happens in my cluster as well. The reasons are mostly concerned with effectiveness:
From my PoV, what @cfis pointed out concerning K8s network is correct.
Example: I have GitLab running, which sends e-mails for CI jobs. GitLab uses my public mail server address It's the same for RoundCube, which accesses ports 465 and 993 inside the cluster as well. Now, the question that was posed reads, "Why not run the traffic through the ingress and use the ProxyProtocol?" This is not easily doable (how would one route the traffic for which the destination IP is already known and on the same host to an intermediate that is not on the same host, just so we can handle it as external traffic)? And we need the traffic to be external in order for it to go through the ingress. What seems to make sense to me now: ┌───────┐
│ │ <── cluster-internal traffic port 465/587 (not proxied) (e.g., for GitLab)
│ │ <── cluster-internal traffic port 993/995 (not proxied) (e.g., for RoundCube)
│ │
│ DMS │ ┌───────────┐
│ │ │ │ <── cluster-external traffic port 25 (proxied)
│ │ <── │ Ingress │ <── cluster-external traffic port 465/587 (proxied)
│ │ │ │ <── cluster-external traffic port 993/995 (proxied)
│ │ └───────────┘
└───────┘
|
Ok, but this is Your description is only conveying that the client IP is preserved to roundcube though, which wasn't my question. It was that roundcube continues to preserve that client IP information when connecting to DMS on behalf of that user, which AFAIK was why they added ProxyProtocol support? What does DMS log for the client IP when you authenticate through Roundcube? Is DMS getting the client IP or only the IP of the Roundcube container?
Are you saying it's not possible for k8s to route through the proxy container? Because I just showed that you can do it with just Docker, which was between two containers in the same network. It should be no different with a reverse proxy container in the middle, I see no reason for that to differ in k8s. I lack the k8s experience to say otherwise, but I'm skeptical of such a limitation. If you believe it really isn't possible, I can ping some maintainers with k8s experience for their input.
No? If it doesn't do that with your expectation of container to container, why do you think that is different when all that is changing is the IP it connects to is a different containers IP within the same subnet?
Docker Compose has services connected to networks Docker creates and manages with it's own internal DNS service for resolving links between containers/services. Not familiar with pods (I know Docker Swarm deals with replicas of a container for scaling, but I've never bothered with Swarm).
Ok.. so in Docker Compose, or just plain Docker, containers are assigned an IP within the subnet they belong to. Assigning an IP to a container explicitly is opt-in. Pods from what I've read represents a group of IPs that are load balanced for a DNS record the cluster manages, unless you're using a headless service or something along those lines 🤷♂️ I had a read over a variety of the docs networking pages, and some other sites, but it seems k8s adds friction for what I proposed.
Traffic shouldn't have to leave the cluster, this isn't what I meant with Docker Compose, just routing to the container that handles reverse proxy and letting that resolve it. Apparently this is quite different distinction/separation from the cluster with ingress role in k8s 🤔 I'd suggest a 2nd instance of the proxy for internal usage if that were the case, but that doesn't help with k8s having less flexibility in the internal routing. I did come across this on StackOverflow where one of the answers to get a network alias for a service is: apiVersion: v1
kind: Service
metadata:
name: "my-alias"
spec:
type: ExternalName
externalName: "other-service.my-namespace.svc.cluster.local"Which lets you access Another suggestion is bundling the containers into a single pod where they can reach each other via |
I am not familiar enough with k8s, see the example I gave with a simple alpine container routing to a different container to send mail via I was only suggesting a common flow for all traffic (internal or external) via the reverse proxy service. That's apparently not simple to achieve in k8s. Not that we would, but if DMS bundled Traefik... then enabling that would proxy all the ports with ProxyProtocol, and it wouldn't matter what you connect (ingress or internal traffic) as that'd be compatible for both scenarios. You'd just need to configure Traefik to trust the ingress IP/subnet for ProxyProtocol. I have heard of side-car containers with k8s, but I suppose it can't be used in that manner 🤷♂️ Regardless, I still see it as two separate features to implement that compliment each other for those that need the workaround (k8s, similar to it's need for hostname workaround). |
|
After all of this, I think my current setup that just uses a load-balancer to open a port directly is the most straight-forward 😂 |
|
@georglauterbach since you have k8s and CoreDNS experience, can you have a quick skim over this article and let me know if that's still valid?
Just trying to grok core blocker with routing with k8s that requires the separate duplicated ports for internal use. I am not familiar with Traefik and Kubernetes well enough, but curious if their Gateway API support allows for similar internal routing or proxying? For @cfis to handle duplicate ports as an alternate workaround, there is a little bit more change than below, but I have a simpler Postfix That'll be part of a revised docs page PR 👍 Reproduction with Docker Composeservices:
reverse-proxy:
image: docker.io/traefik:2.10
networks:
default:
ipv4_address: 172.16.42.10
command:
- "--providers.docker"
- "--providers.docker.exposedbydefault=false"
- "--providers.docker.network=dms-test-network"
# Postfix:
- "--entryPoints.smtp.address=:25"
- "--entryPoints.smtp-submission.address=:587"
- "--entryPoints.smtp-submissions.address=:465"
# Dovecot:
- "--entryPoints.imap.address=:143"
- "--entryPoints.imaps.address=:993"
- "--entryPoints.sieve.address=:4190"
# CAUTION: Production usage should configure socket access better (see Traefik docs)
volumes:
- /var/run/docker.sock:/var/run/docker.sock
ports:
- "25:25"
- "587:587"
- "465:465"
- "143:143"
- "993:993"
dms:
image: mailserver/docker-mailserver:edge
container_name: dms-mta
hostname: mail.example.test
environment:
SSL_TYPE: manual
SSL_CERT_PATH: /tmp/tls/cert.pem
SSL_KEY_PATH: /tmp/tls/key.pem
# NOTE: `configs` used for convenience of reproduction,
# usually these would be split into separate files and used as volume mounts
configs:
- source: dms-accounts
target: /tmp/docker-mailserver/postfix-accounts.cf
- source: overrides-dovecot
target: /tmp/docker-mailserver/dovecot.cf
- source: overrides-postfix-master
target: /tmp/docker-mailserver/postfix-master.cf
- source: tls-cert
target: /tmp/tls/cert.pem
- source: tls-key
target: /tmp/tls/key.pem
labels:
- "traefik.enable=true"
# Plain-text connection to DMS, which allows for STARTTLS negotiation:
- "traefik.tcp.routers.smtp.rule=HostSNI(`*`)"
- "traefik.tcp.routers.smtp.tls=false"
- "traefik.tcp.routers.smtp.entrypoints=smtp"
- "traefik.tcp.routers.smtp.service=smtp"
- "traefik.tcp.services.smtp.loadbalancer.server.port=25"
- "traefik.tcp.services.smtp.loadbalancer.proxyProtocol.version=2"
- "traefik.tcp.routers.smtp-submission.rule=HostSNI(`*`)"
- "traefik.tcp.routers.smtp-submission.tls=false"
- "traefik.tcp.routers.smtp-submission.entrypoints=smtp-submission"
- "traefik.tcp.routers.smtp-submission.service=smtp-submission"
- "traefik.tcp.services.smtp-submission.loadbalancer.server.port=587"
- "traefik.tcp.services.smtp-submission.loadbalancer.proxyProtocol.version=2"
- "traefik.tcp.routers.imap.rule=HostSNI(`*`)"
- "traefik.tcp.routers.imap.tls=false"
- "traefik.tcp.routers.imap.entrypoints=imap"
- "traefik.tcp.routers.imap.service=imap"
- "traefik.tcp.services.imap.loadbalancer.server.port=143"
- "traefik.tcp.services.imap.loadbalancer.proxyProtocol.version=2"
# Sieve is also STARTTLS according to RFC 5804
# NOTE: Dovecot doesn't appear to config explicit/implicit TLS via setting,
# both would normally be accepted but this isn't possible here with Traefik
# requiring the TCP router to either have TLS disabled or do TLS passthrough.
- "traefik.tcp.routers.sieve.rule=HostSNI(`*`)"
- "traefik.tcp.routers.sieve.tls=false"
- "traefik.tcp.routers.sieve.entrypoints=sieve"
- "traefik.tcp.routers.sieve.service=sieve"
- "traefik.tcp.services.sieve.loadbalancer.server.port=4190"
- "traefik.tcp.services.sieve.loadbalancer.proxyProtocol.version=2"
# These implicit TLS ports additionally configure TLS passthrough
- "traefik.tcp.routers.smtp-submissions.rule=HostSNI(`*`)"
- "traefik.tcp.routers.smtp-submissions.tls.passthrough=true"
- "traefik.tcp.routers.smtp-submissions.entrypoints=smtp-submissions"
- "traefik.tcp.routers.smtp-submissions.service=smtp-submissions"
- "traefik.tcp.services.smtp-submissions.loadbalancer.server.port=465"
- "traefik.tcp.services.smtp-submissions.loadbalancer.proxyProtocol.version=2"
- "traefik.tcp.routers.imaps.rule=HostSNI(`*`)"
- "traefik.tcp.routers.imaps.tls.passthrough=true"
- "traefik.tcp.routers.imaps.entrypoints=imaps"
- "traefik.tcp.routers.imaps.service=imaps"
- "traefik.tcp.services.imaps.loadbalancer.server.port=993"
- "traefik.tcp.services.imaps.loadbalancer.proxyProtocol.version=2"
# An explicit subnet was only configured to assign an explicit IP to the reverse-proxy container
# Otherwise the Dovecot haproxy_trusted_networks would need to specify a very wide CIDR range
# (172.16.0.0/12 => 172.16.0.0 to 172.31.255.255)
networks:
default:
name: dms-test-network
ipam:
config:
- subnet: "172.16.42.0/24"
configs:
# DMS requires an account to complete setup
# NOTE: All accounts are configured with the same `secret` password.
dms-accounts:
# NOTE: `$` needed to be repeated to escape it, which opts out of the `compose.yaml` variable interpolation feature.
content: |
john.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
jane.doe@example.test|{SHA512-CRYPT}$$6$$sbgFRCmQ.KWS5ryb$$EsWrlYosiadgdUOxCBHY0DQ3qFbeudDhNMqHs6jZt.8gmxUwiLVy738knqkHD4zj4amkb296HFqQ3yDq4UXt8.
# The ECDSA test suite cert (self-signed):
tls-cert:
content: |
-----BEGIN CERTIFICATE-----
MIIBvDCCAWGgAwIBAgIRAOpRrmDNAnhLvxuk42f/s10wCgYIKoZIzj0EAwIwIDEe
MBwGA1UEAxMVU21hbGxzdGVwIHNlbGYtc2lnbmVkMB4XDTIxMDEwMTAwMDAwMFoX
DTMxMDEwMTAwMDAwMFowIDEeMBwGA1UEAxMVU21hbGxzdGVwIHNlbGYtc2lnbmVk
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsfexEnUXqHKaOTHv8GGy9AyIxgWy
EvKZ4KyBeExylOlSj+nBe7AVg5AGMNLAa2ZjMRRUKIAdiW6kLN1ZF1+mPqN8MHow
DgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAd
BgNVHQ4EFgQUFJ+66xRSnywTsunxd9pb9uIdVL0wKgYDVR0RBCMwIYIMZXhhbXBs
ZS50ZXN0ghFtYWlsLmV4YW1wbGUudGVzdDAKBggqhkjOPQQDAgNJADBGAiEA+HI9
kH7bnnpPbYD7+txCQ+Lcj2rsGMWv4eoD/WZ4ogECIQD3hliBF/V0WxTID5Piu7jp
kEfA97Fggtm0Gvz6ZvpIXA==
-----END CERTIFICATE-----
tls-key:
content: |
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIO4nGpEVFeDjKsUKBBumdJxg0tOx/BEasG6G/denFif1oAoGCCqGSM49
AwEHoUQDQgAEsfexEnUXqHKaOTHv8GGy9AyIxgWyEvKZ4KyBeExylOlSj+nBe7AV
g5AGMNLAa2ZjMRRUKIAdiW6kLN1ZF1+mPg==
-----END EC PRIVATE KEY-----
overrides-dovecot:
content: |
haproxy_trusted_networks=172.16.42.10
# CIDR range is optional, it's better to be more specific about the reverse-proxy IP when possible
#haproxy_trusted_networks=172.16.42.0/24
# Override Dovecot services to use ProxyProtocol:
service imap-login {
inet_listener imaps {
haproxy = yes
}
inet_listener imap {
haproxy = yes
}
}
service managesieve-login {
inet_listener sieve {
haproxy = yes
}
}
overrides-postfix-master:
content: |
smtp/inet/postscreen_upstream_proxy_protocol=haproxy
submission/inet/smtpd_upstream_proxy_protocol=haproxy
submissions/inet/smtpd_upstream_proxy_protocol=haproxyReproduction commands with above config: # Bring up the two containers configured for using PROXY protocol:
$ docker compose up --force-recreate
# Add a third container to that network, with an extra DNS entry to /etc/hosts
# Technically the DNS name is optional as swaks can take an IP directly to --server arg
$ docker run --rm -it \
--add-host dms-internal.example.test:172.16.42.10 \
--network dms-test-network \
alpine:edge
# Add swaks with TLS support:
$ apk add swaks --repository=https://dl-cdn.alpinelinux.org/alpine/edge/testing
$ apk add perl-net-ssleay
# Send a test mail indirectly to DMS with TLS successfully through the reverse-proxy:
$ swaks --server dms-internal.example.test --port 587 -tls \
--from hello@alpine.test --to jane.doe@example.test \
--auth-user john.doe@example.test --auth-password secret
# DMS logs (172.16.42.3 is the Alpine container):
postfix/submission/smtpd[796]: connect from df2cc066c982.dms-test-network[172.16.42.3]
postfix/submission/smtpd[796]: Anonymous TLS connection established from df2cc066c982.dms-test-network[172.16.42.3]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256
postfix/submission/smtpd[796]: AD5C3189D4: client=df2cc066c982.dms-test-network[172.16.42.3], sasl_method=LOGIN, sasl_username=john.doe@example.test
postfix/qmgr[726]: AD5C3189D4: from=<hello@alpine.test>, size=251, nrcpt=1 (queue active)
dovecot: lmtp(jane.doe@example.test)<924><ofLbLcvJwWWcAwAAUi6ngw>: sieve: msgid=<20240206055523.000012@df2cc066c982>: stored mail into mailbox 'INBOX'
postfix/lmtp[923]: AD5C3189D4: to=<jane.doe@example.test>, relay=mail.example.test[/var/run/dovecot/lmtp], delay=0.07, delays=0.05/0/0.01/0.01, dsn=2.0.0, status=sent (250 2.0.0 <jane.doe@example.test> ofLbLcvJwWWcAwAAUi6ngw Saved)Thunderbird client works fine too, although since it's on the same system (Windows via WSL2), the IP used is via the Docker network gateway IP (due to iptables rules for host connections via localhost on published ports), it should be correct if the connection was external. |
|
The coreDNS configuration that you linked seems to be up to date. With this, I can also see how all ports can be PROXY-Protocol-enabled. |
Awesome, @cfis does that work for you as well? Or is there an inconvenience with the CoreDNS rewrite approach? |
|
CoreDNS - you likely need to modify it anyway if you are using a certificate. For example, in my case: Where Now lets say you create a container inside the cluster that has a proxy and point So yes, you can make it work. And I think it is less issue-prone than bouncing traffic outside the cluster. But it still seems like a lot of work to avoid opening extra standard mail ports that aren't even exposed outside the cluster. |
|
FYI thanks for that docker compose file, it is quite illuminating to see a full example, especially the config section. |
I'm not familiar with the differences with k8s here, you'll need to explain that to me.
The extra work is to preserve the client IP. If you don't care about that from the internal container traffic (which may have been proxied already from the reverse-proxy like with Roundcube), then add the duplicate port config I guess? I was just curious about why it wasn't possible to resolve without duplicating ports in k8s by routing all traffic to As @georglauterbach shared, and my own reproduction with compose in the next comment shows, CoreDNS
|
ReproductionQuite a bit more config, but below we extend the prior reproduction with the following:
Configuring for extra proxy ports (especially with the referenced helm config) doesn't seem much simpler/terser than the CoreDNS with a Corefile, at least in my opinion 😅
Config Additions
services:
dns:
image: coredns/coredns:1.11.1
container_name: mail-dns
networks:
default:
ipv4_address: 172.16.42.9
configs:
- source: dns-config
target: /Corefile
- source: dns-zone
target: /zones/internal.test.zone
# This is just a workaround with the CoreDNS image,
# which introduced an undocumented breaking change with WORK_DIR (Dockerfile)
working_dir: /
# This CoreDNS example config can be copy/pasted as a separate compose config,
# If doing so instead of merging, join the same network this way:
networks:
default:
name: dms-test-network
external: true
configs:
dns-config:
content: |
. {
# Rewrite requests for mail.example.test to rp.internal.test
#rewrite name mail.example.test rp.internal.test
# If wanting to also resolve `example.test`, or any of it's subdomains use regex match type:
# NOTE: `answer auto` for ANSWER rewriting is implicit for `name exact` (`exact` is default type for `name`)
rewrite name regex ^(.*\.)?example\.test.$$ rp.internal.test answer auto
# DNS lookup for dms.internal.test (if rewritten from above, it'd also perform this lookup).
# This represents the internal DNS for the private network with the DMS container
# it is no different to matching the k8s cluster equivalent DNS name.
file /zones/internal.test.zone internal.test
# Alternatively, forward to Docker embedded DNS, matching any hostname configured:
# NOTE: If using file and match is successful there, it is resolved and forward rule is skipped.
forward internal.test 127.0.0.11:53
# Forward other requests not yet answered, which is
# needed for the alpine container to install packages.
forward . 1.1.1.1
log
}
# UPDATE: This isn't required, turns out you can forward to embedded DNS Docker uses for hostnames
#
# Normally this zone is already managed by k8s for you with the .svc.cluster.local addresses,
# To demonstrate equivalent via compose, the alpine container uses CoreDNS to route through the reverse-proxy
dns-zone:
# NOTE: $$ required in content to escape $ from ENV interpolation feature
content: |
$$ORIGIN internal.test.
$$TTL 60
@ SOA internal.test. admin.internal.test. 2024102030 4h 1h 14d 10m
@ NS ns1.internal.test.
; A records
@ A 172.16.42.10
rp A 172.16.42.10
ns1 A 172.16.42.9
services:
# Embedded DNS from Docker can resolve to a container in the network via hostname
# Should be similar to k8s svc.cluster.local assigned addresses
reverse-proxy:
hostname: rp.internal.test
dms:
# Unset the original mail.example.test hostname
hostname: !reset
# k8s can't set DMS FQDN via hostname, it must be configured via ENV instead:
environment:
OVERRIDE_HOSTNAME: mail.example.test
# This service is constrained to only run when the "testing" profile is specified,
# it just starts a container to run the pre-configured command to prove the CoreDNS rewrite feature works.
# Usage: docker compose --profile testing run swaks
swaks:
# Use CoreDNS for resolving mail.example.test internally to the reverse-proxy container
dns: 172.16.42.9
# This hostname will resolve in the network when DMS does a lookup (via embedded DNS) which prevents:
# 450 4.1.8 <hello@alpine.test>: Sender address rejected: Domain not found
# It will also be used by swaks as the default client EHLO, which must be an FQDN for a basic SPF check to pass.
hostname: alpine.test
profiles:
- testing
# Automate steps to get the swaks package ready for testing, inline Dockerfile with HereDoc:
# Alternatively just use DMS as the image for swaks + curl
build:
dockerfile_inline: |
FROM alpine:edge
RUN <<EOF
apk add swaks --repository=https://dl-cdn.alpinelinux.org/alpine/edge/testing
apk add perl-net-ssleay curl doggo
EOF
# `>` folds the multi-line yaml content to single line command
command: >
swaks --server mail.example.test --port 25 -tls
--from hello@alpine.test
--to jane.doe@example.test
--tls-ca-path /tmp/ca-cert.pem
# Optionally add credentials for ports 587/465:
# --auth-user john.doe@example.test --auth-password secret
# NOTE: Also include `--tls-sni mail.example.test` if using an explicit HostSNI rule.
configs:
- source: tls-ca-cert
target: /tmp/ca-cert.pem
# Not necessary, but makes it easier to identify this container by IP in logs when it's static
networks:
default:
ipv4_address: 172.16.42.42
# Override the prior example cert files:
# - Use an ECDSA cert that's been signed by a self-signed CA for TLS cert verification.
# - This cert is only valid for mail.example.test
configs:
# The swaks container will need to reference this file contents for successful verficiation.
# Normally not required as a public CA cert like LetsEncrypt is included in the image.
tls-ca-cert:
content: |
-----BEGIN CERTIFICATE-----
MIIBejCCASGgAwIBAgIQRRIFsycc6tiFqOqcXCHZvDAKBggqhkjOPQQDAjAcMRow
GAYDVQQDExFTbWFsbHN0ZXAgUm9vdCBDQTAeFw0yMTAxMDEwMDAwMDBaFw0zMTAx
MDEwMDAwMDBaMBwxGjAYBgNVBAMTEVNtYWxsc3RlcCBSb290IENBMFkwEwYHKoZI
zj0CAQYIKoZIzj0DAQcDQgAEz2IxYBk9cnhgWR4nE93P2RE2KDKv+ijkDm6rS62i
SQDcbIkJmPr38o3tUFPbz21Pzp0aYZfFgHJeJjRKu8uBjKNFMEMwDgYDVR0PAQH/
BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQEwHQYDVR0OBBYEFN6Qs7lNwbPudwCI
i2nsccQw+fZ/MAoGCCqGSM49BAMCA0cAMEQCID87kOfKgnCOPy5yKrknRqzp4krb
VgK8orKZ5I0QetVzAiByJWS2HKqmwxThZjW/oduQ6klZ+UToY96owLubIQhZhw==
-----END CERTIFICATE-----
tls-cert:
content: |
-----BEGIN CERTIFICATE-----
MIIBxTCCAWqgAwIBAgIQHg296UPzq0FEkJdEzVjrSjAKBggqhkjOPQQDAjAcMRow
GAYDVQQDExFTbWFsbHN0ZXAgUm9vdCBDQTAeFw0yMTAxMDEwMDAwMDBaFw0zMTAx
MDEwMDAwMDBaMBkxFzAVBgNVBAMTDlNtYWxsc3RlcCBMZWFmMFkwEwYHKoZIzj0C
AQYIKoZIzj0DAQcDQgAE9FsAamrKHbgVgIHQgnK+rzo8XqebZCEWGSfzdQvr4P5H
amye19qADhsJdkX+i/15CfcIIhqTICF0XniRU0WecaOBkDCBjTAOBgNVHQ8BAf8E
BAMCB4AwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBBTY
vlZSJ+eQsCFbX3nY+NSFV/ArvDAfBgNVHSMEGDAWgBTekLO5TcGz7ncAiItp7HHE
MPn2fzAcBgNVHREEFTATghFtYWlsLmV4YW1wbGUudGVzdDAKBggqhkjOPQQDAgNJ
ADBGAiEArQh78IJBLg7NK/eV/e5z2ZONdHzvKU3V2jME8LaxaxMCIQDX8ZXbvhi4
23e5VwfmuVo9ADTT9esYZ5u6v4hicunJmQ==
-----END CERTIFICATE-----
tls-key:
content: |
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIMl9lvsiSWUqCQ+cvXdiMApfVhQ/XH8zuiNhFD3JbGv9oAoGCCqGSM49
AwEHoUQDQgAE9FsAamrKHbgVgIHQgnK+rzo8XqebZCEWGSfzdQvr4P5Hamye19qA
DhsJdkX+i/15CfcIIhqTICF0XniRU0WecQ==
-----END EC PRIVATE KEY-----CLI commands# compose.yaml + compose.override.yaml are used (Reverse Proxy + DMS):
docker compose up -d --force-recreate
# Bring up CoreDNS:
docker compose -f compose.dns.yaml up -d
# Demonstration with an internal container connecting to DMS indirectly with CoreDNS
# Certificate is verified successfully
docker compose --profile testing run swaksThe log output from # Querying CoreDNS should return an answer with the original DNS query and the expected reverse-proxy container IP
$ docker compose --profile testing run swaks doggo mail.example.test
NAME TYPE CLASS TTL ADDRESS NAMESERVER
mail.example.test. A IN 600s 172.16.42.10 127.0.0.11:53 |
|
In case it's of any additional value, a webserver variant with Caddy and services:
# These must be appended to the main config,
# command is special and gets replaced instead of appended for overrides.
reverse-proxy:
command:
- "--entrypoints.web.address=:80"
- "--entryPoints.websecure.address=:443"
# This relies on configs defined previously for TLS cert files
web:
image: caddy:2.7
labels:
- traefik.enable=true
# The HTTP service type lacks ProxyProtocol support, use TCP as Caddy is configured expecting ProxyProtocol connection.
# Could alternatively have Traefik handle the redirect to 443:
# https://doc.traefik.io/traefik/middlewares/http/redirectscheme/
- traefik.tcp.routers.web.rule=HostSNI(`*`)
- traefik.tcp.routers.web.entrypoints=web
- traefik.tcp.routers.web.service=web
- traefik.tcp.services.web.loadbalancer.server.port=80
- traefik.tcp.services.web.loadbalancer.proxyProtocol.version=2
# NOTE: Specifying an actual SNI will enforce using a TLS cert managed by Traefik:
# SNI matching relies on TLS cert, thus Traefik would intervene even with passthrough.
- traefik.tcp.routers.websecure.rule=HostSNI(`*`)
- traefik.tcp.routers.websecure.entrypoints=websecure
- traefik.tcp.routers.websecure.service=websecure
- traefik.tcp.routers.websecure.tls.passthrough=true
- traefik.tcp.services.websecure.loadbalancer.server.port=443
- traefik.tcp.services.websecure.loadbalancer.proxyProtocol.version=2
configs:
- source: caddyfile
target: /etc/caddy/Caddyfile
- source: tls-cert
target: /dms/certs/cert.pem
- source: tls-key
target: /dms/certs/key.pem
configs:
caddyfile:
content: |
# Global options:
{
local_certs
# Enable ProxyProtocol for incoming Traefik connections
# https://caddyserver.com/docs/caddyfile/options#listener-wrappers
servers {
listener_wrappers {
proxy_protocol {
allow 172.16.42.10/32
}
# Restore default wrappers:
# http_redirect must be added here for redirect support,
# the setting `auto_https disable_redirects` cannot override it.
http_redirect
tls
}
}
}
mail.example.test {
tls /dms/certs/cert.pem /dms/certs/key.pem
respond "Hello {client_ip}!"
}
# These two have an implicit `tls internal` from the `local_certs` global option.
# Caddy will provision with self-signed certs.
hello.example.test {
respond "No problem with rewrites! {client_ip}"
}
rp.internal.test {
respond "The reverse proxy hostname, Client IP: {client_ip}"
}# Cert verifies successfully:
$ docker compose --profile testing run swaks curl -v --cacert /tmp/ca-cert.pem https://mail.example.test -w '\n'
# Alternative FQDN still matches in Caddy via Traefik and Caddy self-signed cert is identified (but cannot verify without CA file)
$ docker compose --profile testing run swaks curl -v --insecure https://hello.example.test -w '\n'
# Same as previous but with different FQDN and shows HTTP redirect to HTTPS working
$ docker compose --profile testing run swaks curl -L --insecure http://rp.internal.test -w '\n' |
Just my 2 cents on this: i do not recall it exactly (think that was somewhere around 2021) but in that time i build a test system to work on one of the issues that were opened asking about how to use proxy protocol. Iirc using port As you already wrote: whomever contributed it was probably mistaken. But unless testing it - to make sure i will not break something we do not have on the radar right now - i wouldn't remove it. edit: forgive me for not reading all the k8s related comments |
I guess this is what this fundamental debate is about: I think on many smaller clusters, preserving the origin IP makes not a lot of sense for cluster-internal traffic as the origin IP is already a local IP anyway. I have whitelisted this traffic already. |
|
Origin IP - I am not seeing the issue. If you open two sets of ports - one with proxy support and one without - then the original client address is maintained for both internal and external clients. You don't need a proxy to track internal client IPs, they are talking directly to docker-mailserver so it already knows their IP addresses. Client IP is important for external connections, which gets preserved by the external proxy (if configured) that is not inside the cluster. The only reason to go through a Proxy for internal clients, as far as I can tell, is to avoid having to open two sets of ports (so 993 without proxy support and 10993 with proxy support). But that requires installing a proxy inside the cluster and configuring it - which is way, way more effort than simply opening two ports per protocol (IMAPS, Submission, etc). |
|
TL;DR:
What about your external connections that route to an internal container, and then connect to DMS where the client IP should have been preserved? Or internal container connections with multiple hops where the client IP of the original container should have been preserved?
Roundcube was brought up before where you don't want the roundcube container IP, you want the IP of the user authenticating through it through their browser. For an internal equivalent, a separate container that provides an API for sending mail (eg via HTTP REST) that multiple internal containers may call instead of having direct SMTP capabilities. One of those containers may be client facing allowing users to provide their own credentials, which introduces the same issue if the API container is always the source IP to DMS; all services relying on it then get locked out by Fail2Ban (alternatively by trusting the API container IP due to local private network, you allow external users to abuse the trust to auth repeatedly without consequence). Perhaps there are other examples, the point was those concerns are better alleviated by preserving the client IP properly and avoiding any misconfiguration issues by keeping it seamless throughout infrastructure, no need to worry about differences in FQDN or ports. At least IMO that consistency of all traffic via single ports and either all connections to through the same route (either direct, or via a proxy when present) is preferable. Clearly we disagree, mostly because of differences with how k8s does things, and the only reason exceptions are made for it unlike others like Podman is because of maintainer interest. ComparisonThis is based on the current state, without any DMS feature support implemented.
Route local traffic through proxyMinimal DNS service configuration (or addition if you already have one): # Whatever the equivalent is in k8s config:
# https://kubernetes.io/docs/tasks/administer-cluster/coredns/#what-s-next
# https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/
services:
dns:
image: coredns/coredns:1.11.1
working_dir: /
configs:
- source: dns-config
target: /Corefile
configs:
dns-config:
content: |
. {
rewrite name mail.example.test rp.internal.test
# Slightly different for k8s, use kubernetes rule instead to forward to the cluster DNS
forward internal.test 127.0.0.11:53
forward . 1.1.1.1
}Simpler config overrides (especially Postfix): configs:
overrides-dovecot:
content: |
haproxy_trusted_networks=172.16.42.10
service imap-login {
inet_listener imap {
haproxy = yes
}
inet_listener imaps {
haproxy = yes
}
}
service pop3-login {
inet_listener pop3 {
haproxy = yes
}
inet_listener pop3s {
haproxy = yes
}
}
overrides-postfix-master:
content: |
smtp/inet/postscreen_upstream_proxy_protocol=haproxy
submission/inet/smtpd_upstream_proxy_protocol=haproxy
submissions/inet/smtpd_upstream_proxy_protocol=haproxySeparate proxy ports configuredYour current helm chart approach (ignoring the postfix SMTP port omission): configMaps:
dovecot.cf:
create: true
path: dovecot.cf
data: |
{{- if .Values.proxyProtocol.enabled }}
haproxy_trusted_networks = {{ .Values.proxyProtocol.trustedNetworks }}
{{- if and (.Values.deployment.env.ENABLE_IMAP) (not .Values.deployment.env.SMTP_ONLY) }}
service imap-login {
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
inet_listener imap_proxy {
haproxy = yes
port = 10143
ssl = no
}
inet_listener imaps_proxy {
haproxy = yes
port = 10993
ssl = yes
}
}
{{- end -}}
{{- if and (.Values.deployment.env.ENABLE_POP) (not .Values.deployment.env.SMTP_ONLY) }}
service pop3-login {
inet_listener pop3 {
port = 110
}
inet_listener pop3s {
port = 995
ssl = yes
}
inet_listener pop3_proxy {
haproxy = yes
port = 10110
ssl = no
}
inet_listener pop3s_proxy {
haproxy = yes
port = 10995
ssl = yes
}
}
{{- end -}}
{{- end -}}
user-patches.sh:
create: true
path: user-patches.sh
data: |
#!/bin/bash
{{- if .Values.proxyProtocol.enabled }}
# Make sure to keep this file in sync with https://github.com/docker-mailserver/docker-mailserver/blob/master/target/postfix/master.cf!
cat <<EOS >> /etc/postfix/master.cf
# Submission with proxy
10587 inet n - n - - smtpd
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_type=dovecot
-o smtpd_reject_unlisted_recipient=no
-o smtpd_sasl_authenticated_header=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
-o smtpd_sender_restrictions=\$mua_sender_restrictions
-o smtpd_discard_ehlo_keywords=
-o milter_macro_daemon_name=ORIGINATING
-o cleanup_service_name=sender-cleanup
-o smtpd_upstream_proxy_protocol=haproxy
# Submissions with proxy
10465 inet n - n - - smtpd
-o syslog_name=postfix/submissions
-o smtpd_tls_wrappermode=yes
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_type=dovecot
-o smtpd_reject_unlisted_recipient=no
-o smtpd_sasl_authenticated_header=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
-o smtpd_sender_restrictions=\$mua_sender_restrictions
-o smtpd_discard_ehlo_keywords=
-o milter_macro_daemon_name=ORIGINATING
-o cleanup_service_name=sender-cleanup
-o smtpd_upstream_proxy_protocol=haproxy
EOS
{{- end }}In fairness, your Postfix configuration can be simpler like this: #!/bin/bash
# Duplicate the config for the submission(s) service ports (587/465) with adjustments for the proxy ports (10587/10465) and syslog_name setting:
postconf -Mf submission/inet | sed -e s/^submission/10587/ -e 's/submission/submission-proxyprotocol/' >> /etc/postfix/master.cf
postconf -Mf submissions/inet | sed -e s/^submissions/10465/ -e 's/submissions/submissions-proxyprotocol/' >> /etc/postfix/master.cf
# Enable ProxyProtocol support for these new service variants:
postconf -P 10587/inet/smtpd_upstream_proxy_protocol=haproxy
postconf -P 10465/inet/smtpd_upstream_proxy_protocol=haproxy
# Create a variant for port 25 too (NOTE: Port 10025 is already assigned in DMS to Amavis):
postconf -Mf smtp/inet | sed -e s/^smtp/12525/ >> /etc/postfix/master.cf
# Enable ProxyProtocol support (different setting as port 25 is handled via postscreen), optionally configure a syslog name to distinguish in logs:
postconf -P 12525/inet/postscreen_upstream_proxy_protocol=haproxy 12525/inet/syslog_name=smtp-proxyprotocolConclusion
I will update the relevant docs page with config advice shared here, it'll be part of the v14 release. If someone wants to implement this feature for v15, you have the improved approach above.
|
No need to read anything else, it's a long thread 😅
If you were the original contributor of the current PROXY protocol docs, then you probably encountered the above. As you mentioned there would otherwise be a conflict with 993 as enabling proxy support rejects any connection that doesn't provide the proxy headers. The docs page will still clearly cover both needs 👍 with proxy support being simplified in a future DMS release. |
Does Roundcube actually do that? I had no idea... I see no real use-case for routing to another container first, which then connects to DMS. Even if there was, the traffic is still internal to the cluster. Only in zero-trust architectures would this matter from a security perspective, right? |
AFAIK, I linked to a thread in our discussions page where a user was having trouble getting it setup. (EDIT: I don't see the link anywhere above, so here it is) There is a plugin for roundcube to support a way via IMAP that Dovecot supports, while the official roundcube support is only partial for PROXY protocol apparently, they can only use it with STARTTLS from what I saw.
I gave one didn't I? A container that provides an HTTP API to send mail through SMTP submission to DMS. If you have another container that's public facing, or routes the HTTP API, you'd have a similar scenario wouldn't you with user authentication if you were building something like SendGrid? (which has an HTTP API for sending mail) That'd need to support preserving the client IP itself, similar to how the Caddy example config demonstrates, but since I don't think Traefik helps in manipulating the PROXY protocol, your API service would need to handle proxying the TCP connection itself (so that client IP info is prepended via PROXY header prior to the SMTP connection itself), either to Traefik or directly to DMS. For rust with Lettre, that seems like you'd need to do your own custom transport I think 🤷♂️ Doesn't seem like it'll be a common requirement, and honestly if it's for something like Fail2Ban, then I guess rather than rely solely on DMS, it'd be better to monitor logs from that API service instead.
I guess? I only have the Roundcube example as a real-world one that I'm aware of where this type of issue could happen unexpectedly, and that perhaps there are other services that could encounter it too. Routing through a proxy won't magically fix that, but it is better to be restrictive on the trusted proxies (where PROXY protocol is established from), since you allow anything on the subnet capable of connecting to forge this connection info otherwise. Given how we have several networking surprises with Docker (and I recall some of these at least applied to k8s in the past, possibly resolved since) and not just IPv6 related, I'm not one to advise trusting the entire subnet so easily just because it's in the private range so assume it's safe 🤷♂️ In fact mailcow docs have a big warning about an IPv6 change to be aware of with recent Docker v25 and backports. With k8s at least, I know that they use These can come/go over time, better to minimize trust. |
|
Based on this discussion, there are 3 types of clients:
Onto Roundcube. From my testing, by default, docker-mailserver sees the source ip as roundcube's ip. For imap, and only imap, you can install a plugin that uses an imap specific feature to send the client ip (which Roundcube knows via HTTP headers). Examples include https://gitlab.com/takerukoushirou/roundcube-dovecot_client_ip and https://gitlab.com/takerukoushirou/roundcube-dovecot_client_ip. Or you can use Roudcube's limited proxy support as mentioned above. However, for ports 465 or 587 (and 110), from what I saw, there is no proxy support. So Roundcube requires having non-proxy ports available - so now you are back to having two sets of ports. So unless I am missing something, that means the only safe setup, at least for Roudcube (or NexctCloud email or SnappyMail or RainLoop) would be requiring authentication both for sending and retrieving emails. If the user can authenticate, then the user is presumably trusted, and the source ip is irrelevant. That isn't any different than other web email clients. For example, if you send yourself an email from Google mail the source ip is Google's server, not my browser's ip address. |
IIRC the issue was related to Fail2Ban within DMS monitoring logs, such as those with IPs that were failing auth multiple times. This is to prevent malicious users trying to access accounts that are not theirs, or DDoS related AFAIK. Without any other known concerns, I'm fine with that being a documented gotcha and users should instead trust those internal IPs and have the other containers auth logs monitored instead.
It never was about receiving mail with a source IP different from DMS. My other example was authenticated submission, related to the same concern with a single container IP representing multiple clients authenticating in the logs through that container service. Except this would have been for ports 587/465 instead. When other mail servers connect to DMS, it's important that their IP is correct, but that's only a known issue with external connections through a proxy which the PROXY protocol support resolves, so we're good there.
This doesn't change. You can route through the reverse-proxy or use a direct connection (by duplicating the ports), your choice. |
|
Fail2Ban - yes - I think you are right. I have no better ideas.
So I realize I don't follow this part. Let's stick with the Roudcube example. To preserve the ip, Roudcube would need to turn on its proxy protocol support (at least for IMAP) when talking to the proxy which would also need to be expecting the Proxy protocol. But then the proxy isn't actually doing anything useful, roundcube could just talk directly to docker-mailserver. If instead Roudcube is not using the proxy protocol and goes through the proxy, then the client ip is going end up being Roundcube's ip address. Anyway, after all this long discussion, have we come to any conclusions? |
I was stating that nothing changed in what you said to require the extra ports be configured. The outcome is the same regardless of DMS having PROXY protocol only ports (where Roundcube would route through the reverse-proxy) or duplicate ports with PROXY protocol enabled on them (so Roundcube could talk directly to DMS instead).
..Yes? See my earlier comment: #3866 (comment) There's a brief TL;DR and at the end of the comment a longer "Conclusion" section with more info regarding that. I will document what was discussed here for v14, and encourage a contribution for duplicate port approach for v15+ |
|
I was finally able to get into Proxy Protocol myself. After enormous other hassles this week (unrelated to DMS itself), I started fiddling around with Proxy Protocol. I set up Traefik according to our docs and used custom ports (10025 (I do not use Amavis, all well), 10465, etc.) for Proxy Protocol traffic. What can I say: it works flawlessly! Now I can finally get rid of my second IPv4 address that I required earlier due to the need for matching A and PTR record and save a few bucks. I run a completely custom
|
polarathene
added
the
stale-bot/ignore
Context
As discussed on the kubernetes page, PROXY protocol support is quite important for Kubernetes.
See the
docker-mailserver-helmports section.Description
Is there any interest in migrating this code to DMS? Would it be useful with Docker too?
What the chart does is create new ports in the 10,000 range that support the proxy protocol for both Postfix and Dovecot.
docker-mailserver-helmrelated config:Could this be added to DMS? Either always on or controlled by an ENV like
ENABLE_PROXY_PROTOCOL_SUPPORT?Alternatives
Leave PROXY protocol support to just kubernetes.
Applicable Users
All users
Are you going to implement it?
Yes, because I know the probability of someone else doing it is low and I can learn from it.
What are you going to contribute?
See already implemented code in the Helm chart.
The text was updated successfully, but these errors were encountered: