docker-sync support

[bsormagec]

I think docker-sync support will be good for mac because docker has poor performance on osx. Can you add this to devilbox ?

[cytopia]

@bsormagec thanks for pointing it out. Can you direct me to some more documentation about this.

[bsormagec]

Sure,

https://github.com/EugenMayer/docker-sync/wiki/2.-Configuration

[vzlatev]

Hello,

Any plans to include it in the upcoming releases - on OSX without docker-sync web apps are awfully slow.

Regards

[coredevel]

+1, without docker-sync there are a couple of options for os x when mounting shares using bind mounts:

• consistent or default: The default setting with full consistency, as described above.
• delegated: The container runtime’s view of the mount is authoritative. There may be delays before updates made in a container are visible on the host.
• cached: The macOS host’s view of the mount is authoritative. There may be delays before updates made on the host are visible within a container.

(Source: https://docs.docker.com/engine/admin/volumes/bind-mounts/#configure-mount-consistency-for-macos)

Unfortunately though none of them appear to resolve the bi-directional sync without huge performance overheads (in the case of the default consistent setting) or persistence issues (e.g. if uploading images to a cms and building up a site's content locally with the docker container instance). Using vagrant in the past for large Magento builds I tried both unison and gatling-rsync, the latter being more performant and the final solution chosen which worked wonders. It ended up being a v similar setup to the docker-sync native_osx implementation, more reading material here:

• http://docker-sync.io/
• https://github.com/EugenMayer/docker-sync
• https://github.com/EugenMayer/docker-sync/wiki/How-does-native_osx-work-with-OSXFS-and-is-still-native-speed-fast%3F

As an aside a short term workaround to resolve performance issues at the moment is to use memcache/redis caching which works really well, managed to get a basic WordPress instance sub 80ms fully rendered in the browser!

[mad99]

Any news about a well usable and reasonably fast system under osx?

Have tried to extend the Makefile on my own (process to include docker-sync-stack). But it does not work well...
Maybe someone else is also to try to do this way. Would great to share or hear from news in this directions!

[cytopia]

This is still on the agenda, however the roadmap is currently really filled and me alone working on that takes some time. Probably somebody wants to jump in on this issue and submit a PR.

[johnwbaxter]

I tried setting up docker-sync but ended up somehow making docker use 400% cpu...will try again though as it's almost unusable on OSX as it stands.

[johnwbaxter]

@cytopia What would it cost to sponsor you to move this to the top of the list?

[cytopia]

@johnwbaxter I'd rather have someone helping me on this issue, as I dont' have a Mac at the moment. So I cannot get sponsored on this one and make an unreliable promises/guess. But thank you for bringing this up.

[cytopia]

There seems to be a native approach by Docker for volume mounts with the following options:

• consistent
• cached
• delegated

How would one of those strategy perform against the current mount options:

https://docs.docker.com/docker-for-mac/osxfs-caching/#tuning-with-consistent-cached-and-delegated-configurations

Might give it a try. If this does not work out, I will have alook into docker-sync

[cytopia]

@bsormagec @vzlatev @Richdel @mad99 @johnwbaxter

The above suggestion should now be easily achievable via global mount options: https://devilbox.readthedocs.io/en/latest/configuration-files/env-file.html#mount-options

[ericgerhardy]

Has anyone solved this problem by using docker-sync or the global mount options? If yes, could you please tell us your configuration? Thanks in advance!

[cytopia]

@ericfynnmaier have you already tried one of these:

MOUNT_OPTIONS=,consistent
MOUNT_OPTIONS=,cached
MOUNT_OPTIONS=,delegated

Probably ,cached will give you some performance boost. I have no OSX anymore to try. Can you give it a shot and report how well it performs in terms of the empty default

[ericgerhardy]

@cytopia thank you! I added ,cached to the .env and e.g. Wordpress loads way faster!

[vielhuber]

,cached only seems to improve performance on mac, but not windows.
how do windows people work with devilbox in a performant manner?

[cytopia]

@vielhuber I haven't looked into Windows yet for those kinds of performance improvements. Any idea what alternatives are offered for Windows

[vielhuber]

The only tool everybody is mentionning is "docker-sync" (http://docker-sync.io/).
This seems to solve the problems everywhere.
But it seems quite complex to install on Windows.

[cytopia]

OK, didn't know that this also works on Windows.

[cytopia]

Docker Sync Support (PR open)

@bsormagec @vzlatev @Richdel @mad99 @johnwbaxter @ericfynnmaier @vielhuber @jeffwhitfield @anselmdk @krebbi @chiribuc @OksanaRomaniv @akuehn @lesteak @Yame- @Ohrlab

I have created a first working draft including instructions how to get this working.

• PR: WIP: Feature: docker-sync #535

Thanks again to @chiribuc jumping in and providing a working example.

[buderus199]

I have tested docker-sync on my updated windows 10 machine with latest stable docker for desktop. Mainly because of slow magento 2 project.

I'm using latest devilbox from master branch and I have added docker-sync.yml & docker-compose-dev.yml file (from docker-sync branch) to devilbox root folder. I did run docker-sync and get "success Starting Docker-Sync in the background", but can't see any improvements in loading times or deployment (php bin/magento setup:static-content:deploy) for magento 2 project. Probably I'm missing something?

[cytopia]

@buderus199 Did you adjust (uncomment) the sync_strategy according to what is required for Windows?

https://github.com/cytopia/devilbox/pull/535/files#diff-09bf18a28f8967be55bc552a00f967dfR32

[buderus199]

@cytopia yes I have used sync_strategy: 'unison'

This is the output when running docker-sync start from Ubuntu WSL:

/var/lib/gems/2.5.0/gems/docker-sync-0.5.11/lib/docker-sync/update_check.rb:41: warning: Insecure world writable dir /mnt/c in PATH, mode 040777
ok Starting unison for sync devilbox-intranet-sync
doing initial sync with unison
Unison 2.51.2 (ocaml 4.06.1): Contacting server...
Looking for changes
Reconciling changes
Propagating updates
UNISON 2.51.2 (OCAML 4.06.1) started propagating changes at 14:30:48.56 on 29 Apr 2019
[BGN] Copying from /app_sync to /host_sync
[END] Copying
UNISON 2.51.2 (OCAML 4.06.1) finished propagating changes at 14:30:48.56 on 29 Apr 2019
Saving synchronizer state
Synchronization complete at 14:30:48 (1 item transferred, 0 skipped, 0 failed)
real 0m 0.04s
user 0m 0.04s
sys 0m 0.00s
chown ing file to uid 1000

And it stays like this for hours. Not sure if it is normal to stay like this...

[cytopia]

@buderus199 If the docker-sync command stales, its probably something to ask these guys: https://github.com/EugenMayer/docker-sync/

[ClearMarble]

I'm using the PR for this and noticing a definite performance improvement on MacOS. CPU usage seems significantly lighter on page loads.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ClearMarble]

Hey, do you imagine this would still work if I updated to the latest version? Been holding off updating due to this but don't know if anything relevant has really changed.

[pirex360]

Still at this date, the last version of devilbox on mac , using laravel...very slow.

[krebbi]

Same here. It's incredible slow on my Macbook Pro.
I have to restart the Docker Images every few hours, otherwise they are slowing down over time.

[cytopia]

The performance is still an issue with MacOS itself - they haven't improved it yet (since years).
The workaround using docker-sync is still blocked by this missing feature: EugenMayer/docker-sync#649

[science695]

@cytopia Since they seem to have found an intense debate on this setting, would it not be possible to default it to native, and in instructions for macos say uncomment this line from the .env sample?

syncs:
  default-sync:
    sync_strategy: '${SYNC_STRATEGY:native}'

env-sample

# For MacOS using docker-sync, uncomment this line:
# SYNC_STRATEGY=rsync

It seems like the extra step to add 1 line to .env file is much better than being without this feature to fix the MacOS performance problem.

[cube-dan]

@cytopia,

Has your mac situation changed since 2018? You said "... as I dont' have a Mac at the moment."

If you have you developed a devilbox specific docker-sync.yml file for PR #535, I'd love to give it a spin and report back my test results here.

Simply put, I'm volunteering time to help, any way I can, to help get the docker-sync feature off the ground.

Thanks @cytopia for all your amazing work on DevilBox!

-Dan

[FJLessing]

@cytopia or anyone really has there been any movement on this front?

[Lenitr]

Docker on Mac has an experimental feature now that allows directory sharing using a Monterey only function called VirtioFS. I'm currently experimenting with it to see if everything runs ok using this. So far it speeds up the containers significantly, especially on requests with a lot of file reads.

Last week i had some issues with composer packages, giving it another go now.

[Lenitr]

So far so good, haven't encountered any issues anymore since switching.
Running composer, installing npm packages etc. all done in devilbox php container ran without issues (at least for my usage).

I'm running MacOS Montery 12.3.1 on an Intel Macbook with Docker Desktop 4.7.1 (4.7.0 last week).

The feature can be enabled here:

Hopefully more people can try it out and post their feedback, I think it's worth a try (but keep in mind that it's an experimental feature).

[anselmdk]

@Lenitr that sounds awesome. Is that all you needed to do, no settings withing devilbox?

[Lenitr]

@anselmdk Yes, correct. The slowness was not a Devilbox issue, but rather Docker on Mac issue. Sharing the filesystem was very slow, but this new technique seems to be a bit faster. Only changing the setting in Docker desktop sped up the containers for me.

To manage your expectations though, it's still not as fast as running any Docker containers on a Linux host, but it's a nice improvement.