High CPU usage starting at 00:00 when external Library is not available since around v1.100.x

[kladderadeng]

The bug

I can not pin down the exact build, but I think it started with version 1.100.0.
Every night, starting at 00:00, immich_microservices starts to consume 100% CPU of one core and redis-server starts to consume about 50% of a core of my server. This is how it looks the next morning when looking at htop:

Checking the output of docker compose logs, I can see this:

And in my monitoring I can see clearly that it all started at 00:00:

I assume this has something to do with my external library. That one sits on a NAS that shuts down every night at 23:30 and can stay off for days. I know this is not optimal for immich, but I made sure that all processes in immich (automatic rescan of librarys and what not) are off. Also, this was not an issue before version 1.100.x.
I have seen that there is an automatic library rescan job in immich that starts at 00:00, but I made sure that it is still off after updating to version 1.100.x. Currently I do not see any configuration or job that starts at 00:00.
I am pretty sure it has something to do with my external library being offline, but I can not figure out (also by reading through the changelogs) what could have changed in that regard to cause this issue.

Before 1.100.x, it was no problem shutting down the NAS. Immich was still running fine and users were able to upload their photos, without immich and redis consuming so much CPU cycles.

The OS that Immich Server is running on

Ubuntu Server 22.04.4 LT

Version of Immich Server

v1.103.1

Version of Immich Mobile App

n/a

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich-server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    restart: unless-stopped
    user: "${PUID}:${PGID}"
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /media/nas/Pictures:/media/nas/Pictures:ro
    env_file:
      - .env
    #ports:
      #- 2283:3001
    depends_on:
      - redis
      - database
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
      - "traefik.enable=true"
      - "traefik.publish=lan"
      - "traefik.http.routers.immich.entrypoints=https"
      - "traefik.http.routers.immich.rule=Host(`immich.example.com`)"
      - "traefik.http.routers.immich.tls=true"
      - "traefik.http.routers.immich.tls.certresolver=letsencrypt"
      - "traefik.http.routers.immich.middlewares=default@file"
      - "traefik.http.routers.immich.service=immich"
      - "traefik.http.services.immich.loadbalancer.server.port=3001"
      - "traefik.docker.network=docker_trusted"
    networks:
      - docker_trusted

  immich-microservices:
    container_name: immich-microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    restart: unless-stopped
    user: "${PUID}:${PGID}"
    extends:
      file: hwaccel.yml
      service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /opt/immich/reverse-geocoding-dump:/usr/src/app/.reverse-geocoding-dump
      - /media/nas/Pictures:/media/nas/Pictures:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
    networks:
      - docker_trusted

  immich-machine-learning:
    container_name: immich-machine-learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    restart: unless-stopped
    user: "${PUID}:${PGID}"
    volumes:
      - /opt/immich/ml-cache:/cache
    env_file:
      - .env
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
    networks:
      - docker_trusted

  redis:
    container_name: immich-redis
    image: redis:6.2-alpine@sha256:60e49e22fa5706cd8df7d5e0bc50ee9bab7c608039fa653c4d961014237cca46
    restart: unless-stopped
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
    networks:
      - docker_trusted

  database:
    container_name: immich-postgres
    image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    restart: unless-stopped
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - /opt/immich/pgdata:/var/lib/postgresql/data
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
    networks:
      - docker_trusted

  backup:
    container_name: immich-db-dumper
    image: prodrigestivill/postgres-backup-local
    restart: unless-stopped
    user: 70:1000
    env_file:
      - .env
    environment:
      POSTGRES_HOST: ${DB_HOSTNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      SCHEDULE: "30 23 0 * * *"
      BACKUP_NUM_KEEP: 7
      BACKUP_DIR: /db_dumps
      TZ: "Europe/Berlin"
    volumes:
      - /opt/immich/db_dumps:/db_dumps
    depends_on:
      - database
    networks:
      - docker_trusted

networks:
  docker_trusted:
    name: docker_trusted
    external: true

Your .env content

UPLOAD_LOCATION=/opt/immich/library
IMMICH_VERSION=release
TYPESENSE_API_KEY=***redacted***
DB_PASSWORD=***redacted***

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich-postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich-redis

# Self-Defined
TZ=Europe/Berlin
PUID=1000
PGID=1000

Reproduction steps

1. NAS with external Library is on
2. Start immich stack
3. Shut NAS with external Library off
4. At 00:00 CPU utilization will go up to 100% for immich_microservices and to 50% for redis-server.

Relevant log output

No response

Additional information

No response