[Feature]: Support more storage backend

[gromain]

Feature detail

Right now, only a local folder is supported natively. Even though more can be supported through fuse mounted filesystems, but this method has drawbacks (performance hit mainly), so it would be nice to have support for more back-ends eventually.

S3 is an obvious choice that comes to mind and that would support a lot of providers, but there could be others (like WebDAV perhaps).

Platform

Server

[Nonobis]

Minio would be great !

[gromain]

Minio is compatible with S3!

[Nonobis]

@gromain : i know :)

[palitu]

Would this be local storage for thumbnails, and remote storage for files? That would maintain speed for scrolling.

Then you would need to pass a Uri to the app to retrieve the full file for viewing?

[akashkj]

If it allows to have a backup also for the media, that would be awesome.
Like 2 storage backends - s3 and G-drive. If one of them is not reachable, switch to other. This could be defined in a priority list.

[gromain]

In my mind it could have been for all objects, but @palitu you're right that thumbnails could still be local.
And yes, @akashkj, that's how I felt about it. Having two or more storage back-ends possible, that can either work as RAID0 or RAID1 (not exactly RAID, but either mirror or concatenate the available storage).

[tarasis]

I would definitely love to see this (I guess I could use a symbolic link for the upload folder). I have a Hetzner Storage Box with 5TB of space, and would love to use it to store the photos, and leave the thumbnails local to the host running immich. (free Oracle Cloud instance).

That way I have some additionally safety as they take snapshots of the data.

[zackpollard]

In my mind, supporting a well defined platform such as S3-like APIs is a good idea and is something we can look into after the first major release is completed. As for the other idea of having multiple backends that we switch between, in my mind that is the job of the storage layer, not something that immich should natively support. We'll leave this here and probably get around to it at some point in the future!

[palitu]

Really great to hear your desire for an S3 type backend. FYI, i use the local filesystem for most of immich, then a mounted NFS share for the main photo storage. My server doesnt have the local storage for the bulk data (especially with all the video's), but thumbs is more than appropriate for local.

I do think that there are some delays with video playing as a result of this, but i assume in the future it will be improved with this S3 feature.

ta

[Cellivar]

I just started writing this up as a new feature request before discovering this one. I would love to be able to use S3 storage for all blob objects and make a raw disk volume optional. Big +1 to this feature request!

At home I run a Nomad cluster, like a Kubernetes cluster but with less yaml involved. We use CSI-managed iSCSI storage volumes that are mounted directly into running docker images (or other tasks). The preference here is a single writer job and maybe multiple read-only mounts for other volumes. This means that sharing a directory is hard while communicating over the network is easy. Right now the docker-compose files seem to suggest mounting one volume to multiple hosts that all have write access which is a little scary (and hard to do with iSCSI). The performance hit of switching to NFS is very visible on our network.

There's an official TS SDK for all the bells and whistles MinIO provides, but I'd strongly suggest looking at just using the S3 SDK which would allow folks to use any S3-compatible storage backend.

This is of course asking a lot, adding a new storage backend like S3 is completely different than writing to disk 😄 I'm perfectly happy to be patient for this sort of feature.

[athornfam2]

Would like to add that I am interested in S3 storage access as well to offload.

[Zen3515]

I try mounting S3 with rclone and found that immich uses softlink which is not allowed in S3.

I guess this is one of the things to consider when implement this in the future.

Error: ENOSYS: function not implemented, link 'upload/04f8f839-9ee4-47fc-89b1-fb3883cf3008/original/WEB/33809e6c-1f1f-4bec-a8cc-7a0367493e8d.png' -> 'upload/04f8f839-9ee4-47fc-89b1-fb3883cf3008/2022/2022-12-11/hahaha.png'

[nperez0111]

I ran into this same issue while trying to use a webdav mount via davfs2

[NiklasRosenstein]

I've run into the same issue, sadly. If it didn't try to use soft links, I think Immich could already run pretty well using S3 fuse mounted storage!

[jcollie]

Had similar issues trying to run Immich in Azure/Azure Kubernetes. Azure does not really have a good ReadWriteMany solution for sharing volumes between pods since it's using CIFS under the hood.

[carballude]

Same thing using SMB mounts:
ERROR [StorageTemplateService] Error: ENOTSUP: operation not supported on socket, link '<<source>>' -> '<<destination>>'

[nicolopozzato]

Hi, i setup immich on a arm vps on oracle, with ocamel fuse google drive, but I have the same error:

immich_server | [Nest] 1 - 03/01/2023, 12:47:57 AM ERROR [StorageService] Error: ENOSYS: function not implemented, link 'upload/85ff3fb7/original/WEB/408963f5-45f7-40e8-960c-79f36116e086.jpeg' -> 'upload/85ff3fb7/2023/2023-02-25/originalName.jpeg'

So in the folder 2023-02-25 the image is not present but it's only in the original/WEB/....

[cycneuramus]

If I'm reading the source code correctly, the reason Immich currently doesn't support rclone mounted backends is simply because:

• its way of moving files is by relying on the mv library, which:
  • works by calling fs.link (at least on clobber: false, which Immich is currently using), which:
    • creates a hard link of the source file at the destination path, at which:
      • mv will unlink the source file, thus having "moved" the file from source to destination.

Now, the problem is, rclone doesn't support hard links. Hence the move operation fails halfway with the ENOSYS: function not implemented, link ... error.

I don't have a thorough understanding of the way Immich is supposed to work here, but if the point is to move the files and nothing more, it seems to me we could achieve full rclone compatibility—and with it support for countless storage backends—simply by reworking the move command (say, by using fs.rename(sourcePath, desPath); or something).

I've created an issue about this here.

EDIT: PR here

EDIT2: PR is now merged, so all rclone supported remotes should work with Immich probably in the next release.

[kajdo]

great!!!

[pinpox]

Is rclone the way to go for using immich with s3 currently? Are there plans to support s3 natively?

[Koveras-hmm]

webdav plz!

[cycneuramus]

Already supported, at least indirectly, via rclone.

[alkmim]

FYI: I did some tests using rclone and I reported my findings (without a lot of data though) here: #3814.

In a nutshell, it did not work reliably enough for me.

Maybe we should not consider this feature supported (indirectly via rclone)

[david-polak]

I just installed Immich on k8s and connected it to an s3 compatible storage, all hosted on Vultr. It's bit of a kludge setup and took me four hours to debug.

Immich pods - mounted nfs dir -> NFS server pod - mounted s3 dir using s3fs -> s3

The results, well it works. Kinda, if parallelism is reduced to 1 it almost handles uploads of regular photos. There are missed thumbnails here and there and corresponding filesystem errors. Videos are mostly broken, haven't had time to see what exactly is the issue with those apart from ffmpeg errors.

Needs more investigation, these are my initial impressions so far.

[NiklasRosenstein]

I'm running things on a CIFS mount as well and it's pretty smooth sailing. Haven't reached the level of brave required to nest an S3 mount inside it, though.

[floonym]

I am currently using rclone to mount google drive with seemingly no errors using vfs-cache-mode write. I only mounted the library and upload folder of the docker containers from the rclone folder, having the thumbnails mounted from the server locally.

rclone mount drive: /mnt/drive --vfs-cache-mode writes -vv

[janbuchar]

I was thinking about contributing s3 support to immich. I found that

1. the uploads are stored directly to the library by multer - I could probably use multer-s3 to take care of this
2. exiftool needs a local copy of the file, but it shouldn't be a problem to download a temporary copy just for metadata extraction
3. most other filesystem-related functionality is handled by IStorageRepository, it should be enough to make an alternative implementation of that
4. files are served using sendFile from express, and it may be tricky to adjust serveFile to support direct streaming from s3

Am I missing some other roadblocks? Would this level of support for s3 even be welcome?

[pinpox]

Please do, I'm eagerly awaiting this feature. If you need help testing let me know!

[janbuchar]

I'd also love to have it, but I really need some feedback from the team before. I'd hate to spend time on something that wouldn't get merged in the end 🙂

[janbuchar]

It seems that @uhthomas plans to work on this as well - see #4445

[david-polak]

Hmm I threw some hours at it couple of months back... main...david-polak:immich:s3 with no avail, I got basic stuff working but not everything was configured to work with streams or the file access layer, the actual refactor needed to properly support s3 is significant and should be undertaken under the supervision and approval of the project leaders.

[systemmonkey42]

+! for this to support structured storage for original media while keeping thumbnails and cache data on a local filesystem.

This would also benefit from some method to take a database dump and upload it to the same structured storage for future DR scenarios.

Currently I use snapshots for the database, but that not going to help in the event of a fire, even though all the original media is currently backed up to an S3 bucket.