Getting SignatureDoesNotMatch on objectstore reconciliation event

[kayrus]

I'm getting failed to reconcile CephObjectStore rook-ceph/ceph-objectstore-st1-default. failed to create object store deployments: failed to get COSI user cosi: SignatureDoesNotMatch tx00000caa4f7677e4cdac4-0066438484-9cced-ceph-objectstore-st1-default 9cced-ceph-objectstore-st1-default-ceph-objectstore-st1-default while creating an objectstore.

Here is my cephobjectstore resource:

apiVersion: v1
items:
- apiVersion: ceph.rook.io/v1
  kind: CephObjectStore
  metadata:
    annotations:
      meta.helm.sh/release-name: rook-ceph
      meta.helm.sh/release-namespace: rook-ceph
    creationTimestamp: "2024-05-07T13:50:20Z"
    finalizers:
    - cephobjectstore.ceph.rook.io
    generation: 9
    labels:
      app.kubernetes.io/managed-by: Helm
    name: ceph-objectstore-st1-default
    namespace: rook-ceph
    resourceVersion: "3939729"
    uid: 47e9f631-5acf-47c7-9d49-a27547903cac
  spec:
    dataPool:
      application: ""
      deviceClass: hdd
      failureDomain: host
      mirroring: {}
      quotas: {}
      replicated:
        size: 3
      statusCheck:
        mirror: {}
    gateway:
      instances: 6
      placement:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.metal/role
                operator: In
                values:
                - ceph-osd
      port: 80
      priorityClassName: system-cluster-critical
      resources:
        requests:
          cpu: "11"
          memory: 16Gi
      securePort: 443
      sslCertificateRef: ceph-objectstore-extern-wildcard-ec-st1-default
    healthCheck: {}
    metadataPool:
      application: ""
      deviceClass: nvme
      erasureCoded:
        codingChunks: 0
        dataChunks: 0
      failureDomain: zone
      mirroring: {}
      quotas: {}
      replicated:
        replicasPerFailureDomain: 2
        size: 4
        subFailureDomain: host
      statusCheck:
        mirror: {}
    preservePoolsOnDelete: true
    sharedPools:
      dataPoolName: ""
      metadataPoolName: ""
      preserveRadosNamespaceDataOnDelete: false
    zone:
      name: ""
  status:
    endpoints:
      insecure:
      - http://rook-ceph-rgw-ceph-objectstore-st1-default.rook-ceph.svc:80
      secure:
      - https://rook-ceph-rgw-ceph-objectstore-st1-default.rook-ceph.svc:443
    observedGeneration: 5
    phase: Failure
kind: List
metadata:
  resourceVersion: ""

and the config-override:

$ kubectl -n rook-ceph get cm rook-config-override -o yaml
apiVersion: v1
data:
  config: |
    [client]
    rgw keystone api version = 3
    rgw s3 auth use keystone = true
    rgw keystone url = http://identity.local:8080
    rgw keystone verify ssl = true
    rgw keystone implicit tenants = true
    rgw swift account in url = true
    rgw keystone accepted roles = objectstore_admin
    rgw keystone token cache size = 1000
    rgw keystone admin user = ceph
    rgw keystone admin password = XXX
    rgw keystone admin domain = default
    rgw keystone admin project = admin
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: rook-ceph
    meta.helm.sh/release-namespace: rook-ceph
  creationTimestamp: "2024-05-07T13:41:41Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: rook-config-override
  namespace: rook-ceph
  resourceVersion: "1872630"
  uid: 968e9830-a44b-4b2e-92c6-babfd25b4394

accessing S3 and Swift API works fine, the problem is only for cosi user. I also created a small script that tries to retrieve a cosi user using the credentials returned by the radosgw-admin user info --uid rgw-admin-ops-user | jq -r '.keys[]', but it still returns SignatureDoesNotMatch.

CEPH version: v18.2.2

Please advice.

[BlaineEXE]

I've never seen this particular message from radosgw-admin before.

The first thing to do is make sure that the version of Ceph inside the rook-ceph-operator-* pod is equal to or higher than the version of Ceph installed in the cluster. The Ceph RGW team frustratingly does not guarantee forwards compatibility of radosgw-admin, only backwards compatibility.

If the version is lower in the operator pod, you will have to update the Rook version used.

[kayrus]

This error is not from the radosgw-admin, but from the http://radosgw.api/admin/ endpoint.
I'm using radosgw-admin inside the toolbox to fetch access keys for the rgw-admin-ops-user.

The SignatureDoesNotMatch is returned in the operator logs, description of the cephobjectstore, and using a script that consumes admin API.

Accessing http://radosgw.api S3 API or http://radosgw.api/swift/v1 API works fine using the keystone credentials. Could it be the result of using the keystone for the auth?

    [client]
    rgw keystone api version = 3
    rgw s3 auth use keystone = true
    rgw keystone url = http://identity.local:8080
    rgw keystone verify ssl = true
    rgw keystone implicit tenants = true
    rgw swift account in url = true
    rgw keystone accepted roles = objectstore_admin
    rgw keystone token cache size = 1000
    rgw keystone admin user = ceph
    rgw keystone admin password = XXX
    rgw keystone admin domain = default
    rgw keystone admin project = admin

[kayrus]

UPD: I updated the keystone override config and changed [client] to a garbage [client.abcdefgh] - admin API auth started to work, but the Keystone based S3/Swift API is now broken.

I was sure that oncev the Keystone auth doesn't work, Rados must fallback to the internal auth. Isn't this true anymore?

UPD2: I set the [client.rgw] section, but the Keystone based S3/Swift API still doesn't work, while the /admin API with the local auth still works fine. Switching back to [client] breaks /admin API, but restores Keystone based S3/Swift :/

How can I enable both auth? Is there a fallback option that I'm not aware of?

UPD3: I set the config section to [client.rgw.ceph.objectstore.st1.default.a], and admin API still doesn't work 😠

UPD4: I enabled debug logs with the global's debug rgw = 20. with the keystone enabled admin API access gives:

debug 2024-05-15T10:10:26.920+0000 7f23d2c76700 20 req 6248828640183869205 0.030001037s :get_user_info rgw::auth::keystone::EC2Engine rejected with reason=-2027
debug 2024-05-15T10:10:26.920+0000 7f23d2c76700 20 req 6248828640183869205 0.030001037s :get_user_info rgw::auth::s3::AWSv2ExternalAuthStrategy rejected with reason=-2027
debug 2024-05-15T10:10:26.920+0000 7f23d2c76700 20 req 6248828640183869205 0.030001037s :get_user_info rgw::auth::s3::AWSAuthStrategy rejected with reason=-2027
debug 2024-05-15T10:10:26.920+0000 7f23d2c76700  5 req 6248828640183869205 0.030001037s :get_user_info Failed the auth strategy, reason=-2027
debug 2024-05-15T10:10:26.920+0000 7f23d2c76700 10 failed to authorize request

with the keystone disabled:

debug 2024-05-15T10:15:23.785+0000 7f234cf2e700 20 req 2484819406784737633 0.001000034s :get_user_info rgw::auth::s3::LocalEngine granted access
debug 2024-05-15T10:15:23.785+0000 7f234cf2e700 20 req 2484819406784737633 0.001000034s :get_user_info rgw::auth::s3::AWSAuthStrategy granted access

So it looks like the keystone auth disables the rgw::auth::s3::LocalEngine.

[kayrus]

I discovered the rgw_s3_auth_order option and it looks like setting it to local,external solves the auth problem:
https://github.com/ceph/ceph/blob/4a592ad7fe8914d62131da5bb4259c61bba82a1d/src/common/options/rgw.yaml.in#L883-L894
I wonder why it's not listed in docs: https://docs.ceph.com/en/latest/radosgw/config-ref/

[BlaineEXE]

It seems like this is resolving the issue, yes? If not, I think we will have to contact some RGW devs, because I'm not familiar with these internals.

[kayrus]

I think this is just a workaround. I really want to know why there is no fallback to the local auth method.

[BlaineEXE]

I think the best thing to do from here is for you to join the Ceph slack channel, and try describing your situation there. You could probably link to this discussion. And feel free to @ me there as well.