🐛Cannot fix Docker cloudflared container logs error "Failed to sufficiently increase send buffer size"

[mackonsti]

Describe the bug
Running cloudflared via Docker produces an error in the logs but still runs fine:
Failed to sufficiently increase send buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details

How To Reproduce
Steps to reproduce the behavior:

1. Install the Docker cloudflared image via a simple compose file
2. Launch the container
3. See error in the logs
4. cloudflared config (compose.yml)

version: "3.9"
services:
  tunnel:
    container_name: cloudflared
    image: cloudflare/cloudflared:latest
    restart: always
    command: tunnel run
    environment:
      - TUNNEL_TOKEN=<REMOVED>

Expected behavior
This error should not be displayed, at least not when using Docker cloudflared image, I think.

Environment and versions

• OS: OpenMediaVault 6.9.13-1 (Debian 11)
• Architecture: Intel x64
• Version: Latest cloudflared image (the issue has been there for sometime)

Logs and errors
Despite the fact that the Docker image/container works fine, I get this error in logs:
Failed to sufficiently increase send buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.

Additional context
I have read carefully the Wiki in question and have increased these buffers on the actual OMV (Debian) server side by adding the right parameters in /etc/sysctl.conf that survive reboot and report the correct intended/set values. However this doesn't seem to pass to Docker and I get still the error above.

net.core.rmem_max=26214400
net.core.rmem_default=26214400

Moreover, I tried adding sysctls to my compose file, so that these are set on Docker-level, but the specific net.core.rmem parameters are not accepted, an error is reproduced.

Raised this in the OMV forum with no results or fix possible on the server-level.
Thank you.

[Dezrin]

Experiencing the same issue.

I followed your previous posts on openmediavault and then ended up here, in the hopes you'd had some further feedback.

I attempted to follow the /etc/sysctl.conf fix, but upon checking the RedHat Documentation, their example for increasing the socket recieve buffer size includes 2 parameters that you hadn't stated:

net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=262144
net.core.wmem_max=262144

I'm not sure if you applied these or not? However, since making these changes and restarting my container - I am no longer able to reproduce the error within the logs.

Since applying these changes, I have noticed a significant increase in Cloudflare Tunnel performance, where services were struggling to be stable previously, they are behaving as if within the local environment. Which is fantastic.

Hope this provides some useful information or help, I'm unsure as to if this resolved the problem or if it's just no longer reporting in the logs.

Environment Details:

• OS: Ubuntu 22.04.3 LTS
• Docker version: 24.0.5, build 24.0.5-0ubuntu1~22.04.1
• Portainer Business Edition 2.19.4
• Docker Command:

docker run --name=cloudflare-tunnel --hostname=<hostname> --user=nonroot --mac-address=<mac> --cap-add=AUDIT_WRITE --cap-add=CHOWN --cap-add=DAC_OVERRIDE --cap-add=FOWNER --cap-add=FSETID --cap-add=KILL --cap-add=MKNOD --cap-add=NET_BIND_SERVICE --cap-add=NET_RAW --cap-add=SETFCAP --cap-add=SETGID --cap-add=SETPCAP --cap-add=SETUID --cap-add=SYS_CHROOT --cap-drop=AUDIT_CONTROL --cap-drop=BLOCK_SUSPEND --cap-drop=DAC_READ_SEARCH --cap-drop=IPC_LOCK --cap-drop=IPC_OWNER --cap-drop=LEASE --cap-drop=LINUX_IMMUTABLE --cap-drop=MAC_ADMIN --cap-drop=MAC_OVERRIDE --cap-drop=NET_ADMIN --cap-drop=NET_BROADCAST --cap-drop=SYSLOG --cap-drop=SYS_ADMIN --cap-drop=SYS_BOOT --cap-drop=SYS_MODULE --cap-drop=SYS_NICE --cap-drop=SYS_PACCT --cap-drop=SYS_PTRACE --cap-drop=SYS_RAWIO --cap-drop=SYS_RESOURCE --cap-drop=SYS_TIME --cap-drop=SYS_TTY_CONFIG --cap-drop=WAKE_ALARM --network=bridge --workdir=/home/nonroot --restart=no --runtime=runc cloudflare/cloudflared:latest tunnel --no-autoupdate run --token <token>

Update

Having attempted the fix in the Production enviroment, and not my Development one, I was able to see the error still present:

Going to attempt the fix on there also, and report back.

Update @ 14:00 09/04/2024

Confirmed that this resolves the problem, and I am no longer experiencing the failure within the logs.

To make the changes without editing the /etc/sysctl.conf file, use the following commands:

sysctl -w net.core.rmem_default=26214400
sysctl -w net.core.wmem_default=26214400
sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.wmem_max=26214400

Restart the container and this error should no longer be present within the logs.
This bug can be resolved.

[ChevySSinSD]

I also have the same issue running Debian 12 with containerd running in docker.

I have verified I increased the buffer sizes in my host:

# sysctl net.core.rmem_default 
net.core.rmem_default = 26214400
# sysctl net.core.rmem_max
net.core.rmem_max = 26214400

I attempted to add sysctls to my docker compose for cloudflared:

  cloudflared:
    container_name: cloudflared
    image: cloudflare/cloudflared
    depends_on:
      traefik:
        condition: service_healthy  # Do not start until traefik is started and healthy
    networks:
      - proxy
    volumes:
      - /home/eric/docker/secrets/tunnelcert.pem:/etc/cloudflared/cert.pem:ro
      - /home/eric/docker/secrets/tunneltoken.json:/tunneltoken.json:ro
    sysctls:
      net.core.rmem_default: 26214400 
      net.core.rmem_max: 26214400
    command: tunnel --credentials-file /tunneltoken.json --loglevel warn run Home

However, I get this error:
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /proc/sys/net/core/rmem_max: no such file or directory: unknown

[Dezrin]

Confirmed that this resolves the problem, and I am no longer experiencing the failure within the logs.

To make the changes without editing the /etc/sysctl.conf file, use the following commands:

sysctl -w net.core.rmem_default=26214400
sysctl -w net.core.wmem_default=26214400
sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.wmem_max=26214400

Restart the container, and the error should no longer be present within the logs.
This bug can be resolved.

[mackonsti]

Hi @Dezrin thank you for your find on the RedHat Documentation, I made those changes in my OMV 6.x server permanent via /etc/sysctl.conf and after rebooting, checked the container logs for both Cloudflared and Syncthing (now also showing this error) and they are no longer reported.

I made a personal mention and thanks to you and your great find, in the respective OMV thread that I had started.

It's interesting that these two added parameters are not mentioned in the original Wiki page that the error prompts us to visit and read about it, wonder if it's limited to some OS'es?

If other people confirm this working, permanent or not (via /etc/sysctl.conf) we can consider this as resolved. Thank you again.

[ChevySSinSD]

I can also confirm that adding the changes to the both rmem and wmem parameters and restarting the container resolved the issue for me with host running Debian 12. Previously, I only updated the rmem parameters, which did not resolve the issue. Thank you!