Upgrade from 2024.2.2 to 2024.4.2 Raises issue on SCIM users primary key.

[gcarrarom]

While upgrading to 2024.4.2 the Server and worker pods fail with the following issue:

  Applying authentik_providers_scim.0007_scimgroup_scim_id_scimuser_scim_id_and_more...

Fixing primary key for SCIM users, this might take a couple of minutes...

Fixing primary key for SCIM groups, this might take a couple of minutes...
{"domain_url": null, "event": "releasing database lock", "level": "info", "logger": "lifecycle.migrate", "pid": 18, "schema_name": "public", "timestamp": "2024-05-13T15:03:23.504225"}
Failed to read config file: ./lifecycle/gunicorn.conf.py
Traceback (most recent call last):
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django_prometheus/db/common.py", line 69, in execute
    return super().execute(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedObject: constraint "authentik_providers_scim_id_group_id_provider_id_9d50d292_uniq" of relation "authentik_providers_scim_scimgroup" does not exist

Rollback to the 2024.2.2 version works just fine.

To Reproduce

1. Upgrade authentik using the helm chart from 2024.2.2 to 2024.4.2.
2. Wait for the Server pod to start running.
3. Watch the logs for the error message.

Expected behavior
I'd expect the upgrade of the app, it'll take care of missing relationships in the Database and ensure that it would work out of the box.

Logs

{"app_name": "authentik.stages.email", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.stages.email.tasks", "pid": 18, "schema_name": "public", "timestamp": "2024-05-13T15:03:21.264478"}
{"app_name": "authentik.core", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.core.tasks", "pid": 18, "schema_name": "public", "timestamp": "2024-05-13T15:03:21.266393"}
{"app_name": "authentik.core", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.core.signals", "pid": 18, "schema_name": "public", "timestamp": "2024-05-13T15:03:21.266523"}
{"app_name": "authentik.enterprise", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.enterprise.tasks", "pid": 18, "schema_name": "public", "timestamp": "2024-05-13T15:03:21.266957"}
{"app_name": "authentik.enterprise", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.enterprise.signals", "pid": 18, "schema_name": "public", "timestamp": "2024-05-13T15:03:21.267415"}
{"app_name": "authentik.enterprise.providers.rac", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.enterprise.providers.rac.signals", "pid": 18, "schema_name": "public", "timestamp": "2024-05-13T15:03:21.271826"}
{"app_name": "authentik.events", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.events.tasks", "pid": 18, "schema_name": "public", "timestamp": "2024-05-13T15:03:21.272119"}
{"app_name": "authentik.events", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.events.signals", "pid": 18, "schema_name": "public", "timestamp": "2024-05-13T15:03:21.272226"}
=== Starting migration
Operations to perform:
  Apply all migrations: auth, authentik_blueprints, authentik_brands, authentik_core, authentik_crypto, authentik_enterprise, authentik_events, authentik_flows, authentik_outposts, authentik_policies, authentik_policies_dummy, authentik_policies_event_matcher, authentik_policies_expiry, authentik_policies_expression, authentik_policies_password, authentik_policies_reputation, authentik_providers_ldap, authentik_providers_oauth2, authentik_providers_proxy, authentik_providers_rac, authentik_providers_radius, authentik_providers_saml, authentik_providers_scim, authentik_rbac, authentik_sources_ldap, authentik_sources_oauth, authentik_sources_plex, authentik_sources_saml, authentik_sources_scim, authentik_stages_authenticator_duo, authentik_stages_authenticator_sms, authentik_stages_authenticator_static, authentik_stages_authenticator_totp, authentik_stages_authenticator_validate, authentik_stages_authenticator_webauthn, authentik_stages_captcha, authentik_stages_consent, authentik_stages_deny, authentik_stages_dummy, authentik_stages_email, authentik_stages_identification, authentik_stages_invitation, authentik_stages_password, authentik_stages_prompt, authentik_stages_source, authentik_stages_user_delete, authentik_stages_user_login, authentik_stages_user_logout, authentik_stages_user_write, authentik_tenants, contenttypes, guardian, sessions
Running migrations:
  Applying authentik_providers_scim.0007_scimgroup_scim_id_scimuser_scim_id_and_more...

Fixing primary key for SCIM users, this might take a couple of minutes...

Fixing primary key for SCIM groups, this might take a couple of minutes...
{"domain_url": null, "event": "releasing database lock", "level": "info", "logger": "lifecycle.migrate", "pid": 18, "schema_name": "public", "timestamp": "2024-05-13T15:03:23.504225"}
Failed to read config file: ./lifecycle/gunicorn.conf.py
Traceback (most recent call last):
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django_prometheus/db/common.py", line 69, in execute
    return super().execute(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedObject: constraint "authentik_providers_scim_id_group_id_provider_id_9d50d292_uniq" of relation "authentik_providers_scim_scimgroup" does not exist

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/ak-root/venv/lib/python3.12/site-packages/gunicorn/app/base.py", line 111, in get_config_from_filename
    spec.loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/lifecycle/gunicorn.conf.py", line 147, in <module>
    run_migrations()
  File "/lifecycle/migrate.py", line 113, in run_migrations
    execute_from_command_line(["", "migrate_schemas"])
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django_tenants/management/commands/migrate_schemas.py", line 63, in handle
    executor.run_migrations(tenants=[self.PUBLIC_SCHEMA_NAME])
  File "/ak-root/venv/lib/python3.12/site-packages/django_tenants/migration_executors/standard.py", line 11, in run_migrations
    run_migrations(self.args, self.options, self.codename, self.PUBLIC_SCHEMA_NAME)
  File "/ak-root/venv/lib/python3.12/site-packages/django_tenants/migration_executors/base.py", line 59, in run_migrations
    migrate_command_class(stdout=stdout, stderr=stderr).execute(*args, **options)
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/operations/models.py", line 659, in database_forwards
    alter_together(
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/base/schema.py", line 594, in alter_unique_together
    self._delete_composed_index(
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/base/schema.py", line 658, in _delete_composed_index
    self.execute(self._delete_constraint_sql(sql, model, constraint_names[0]))
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/postgresql/schema.py", line 48, in execute
    return super().execute(sql, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/base/schema.py", line 201, in execute
    cursor.execute(sql, params)
  File "/ak-root/venv/lib/python3.12/site-packages/sentry_sdk/integrations/django/__init__.py", line 644, in execute
    result = real_execute(self, sql, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django_prometheus/db/common.py", line 69, in execute
    return super().execute(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: constraint "authentik_providers_scim_id_group_id_provider_id_9d50d292_uniq" of relation "authentik_providers_scim_scimgroup" does not exist
{"error":"authentik starting","event":"failed to proxy to backend","level":"warning","logger":"authentik.router","timestamp":"2024-05-13T15:03:23Z"}
{"error":"exit status 1","event":"gunicorn process died, restarting","level":"warning","logger":"authentik.router","timestamp":"2024-05-13T15:03:23Z"}
{"error":"exit status 1","event":"gunicorn failed to start, restarting","level":"error","logger":"authentik.router","timestamp":"2024-05-13T15:03:23Z"}
{"event": "Loaded config", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715612604.3914607, "file": "/authentik/lib/default.yml"}
{"event": "Loaded environment variables", "level": "debug", "logger": "authentik.lib.config", "timestamp": 1715612604.3919697, "count": 72}
{"event": "Starting authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715612604.8555431}
{"event": "PostgreSQL connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715612604.9623818}
{"event": "Redis Connection successful", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715612604.9655454}
{"event": "Finished authentik bootstrap", "level": "info", "logger": "authentik.lib.config", "timestamp": 1715612604.965652}

Version and Deployment:

• authentik version: 2024.4.2
• Deployment: helm

[BeryJu]

Could you try to upgrade to 2024.4.1 and then to 2024.4.2? We test version upgrades as part of our CI pipeline, however we only test upgrades from the immediate previous version

[gcarrarom]

Thanks for the heads-up on that! I've upgraded without issues to 2024.4.1, but from 2024.4.1 to 2024.4.2 it fails with the same issue:

"pid": 32, "schema_name": "public", "timestamp": "2024-05-13T18:40:08.221343"}
{"app_name": "authentik.events", "domain_url": null, "event": "Imported related module", "level": "info", "logger": "authentik.blueprints.apps", "module": "authentik.events.signals", "pid": 32, "schema_name": "public", "timestamp": "2024-05-13T18:40:08.221450"}
=== Starting migration
Operations to perform:
  Apply all migrations: auth, authentik_blueprints, authentik_brands, authentik_core, authentik_crypto, authentik_enterprise, authentik_events, authentik_flows, authentik_outposts, authentik_policies, authentik_policies_dummy, authentik_policies_event_matcher, authentik_policies_expiry, authentik_policies_expression, authentik_policies_password, authentik_policies_reputation, authentik_providers_ldap, authentik_providers_oauth2, authentik_providers_proxy, authentik_providers_rac, authentik_providers_radius, authentik_providers_saml, authentik_providers_scim, authentik_rbac, authentik_sources_ldap, authentik_sources_oauth, authentik_sources_plex, authentik_sources_saml, authentik_sources_scim, authentik_stages_authenticator_duo, authentik_stages_authenticator_sms, authentik_stages_authenticator_static, authentik_stages_authenticator_totp, authentik_stages_authenticator_validate, authentik_stages_authenticator_webauthn, authentik_stages_captcha, authentik_stages_consent, authentik_stages_deny, authentik_stages_dummy, authentik_stages_email, authentik_stages_identification, authentik_stages_invitation, authentik_stages_password, authentik_stages_prompt, authentik_stages_source, authentik_stages_user_delete, authentik_stages_user_login, authentik_stages_user_logout, authentik_stages_user_write, authentik_tenants, contenttypes, guardian, sessions
Running migrations:
  Applying authentik_providers_scim.0007_scimgroup_scim_id_scimuser_scim_id_and_more...

Fixing primary key for SCIM users, this might take a couple of minutes...

Fixing primary key for SCIM groups, this might take a couple of minutes...
{"domain_url": null, "event": "releasing database lock", "level": "info", "logger": "lifecycle.migrate", "pid": 32, "schema_name": "public", "timestamp": "2024-05-13T18:40:10.671378"}
Failed to read config file: ./lifecycle/gunicorn.conf.py
Traceback (most recent call last):
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django_prometheus/db/common.py", line 69, in execute
    return super().execute(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedObject: constraint "authentik_providers_scim_id_group_id_provider_id_9d50d292_uniq" of relation "authentik_providers_scim_scimgroup" does not exist

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/ak-root/venv/lib/python3.12/site-packages/gunicorn/app/base.py", line 111, in get_config_from_filename
    spec.loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/lifecycle/gunicorn.conf.py", line 147, in <module>
    run_migrations()
  File "/lifecycle/migrate.py", line 113, in run_migrations
    execute_from_command_line(["", "migrate_schemas"])
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django_tenants/management/commands/migrate_schemas.py", line 63, in handle
    executor.run_migrations(tenants=[self.PUBLIC_SCHEMA_NAME])
  File "/ak-root/venv/lib/python3.12/site-packages/django_tenants/migration_executors/standard.py", line 11, in run_migrations
    run_migrations(self.args, self.options, self.codename, self.PUBLIC_SCHEMA_NAME)
  File "/ak-root/venv/lib/python3.12/site-packages/django_tenants/migration_executors/base.py", line 59, in run_migrations
    migrate_command_class(stdout=stdout, stderr=stderr).execute(*args, **options)
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/migrations/operations/models.py", line 659, in database_forwards
    alter_together(
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/base/schema.py", line 594, in alter_unique_together
    self._delete_composed_index(
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/base/schema.py", line 658, in _delete_composed_index
    self.execute(self._delete_constraint_sql(sql, model, constraint_names[0]))
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/postgresql/schema.py", line 48, in execute
    return super().execute(sql, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/base/schema.py", line 201, in execute
    cursor.execute(sql, params)
  File "/ak-root/venv/lib/python3.12/site-packages/sentry_sdk/integrations/django/__init__.py", line 644, in execute
    result = real_execute(self, sql, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py", line 103, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/django_prometheus/db/common.py", line 69, in execute
    return super().execute(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/ak-root/venv/lib/python3.12/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: constraint "authentik_providers_scim_id_group_id_provider_id_9d50d292_uniq" of relation "authentik_providers_scim_scimgroup" does not exist