1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set

[stemid]

I had a healthy host-based cluster where I had to expand RAM, and there was no hot-add.

So for one node at the time I scaled down its OSD, drained it, shut it down, expanded RAM, started it back, uncordoned it, scaled up the osd deploy.

This worked for a couple nodes, but after expanding the RAM on the 3rd node and finishing all the steps I see this in ceph status.

  cluster:
    id:     1f1dd814-3ac7-4030-82b8-2748cbc2ca25
    health: HEALTH_WARN
            1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set

  services:
    mon: 3 daemons, quorum b,d,e (age 17m)
    mgr: a(active, since 12M)
    osd: 9 osds: 9 up (since 17m), 9 in (since 15m); 35 remapped pgs

  data:
    pools:   2 pools, 129 pgs
    objects: 131.15k objects, 509 GiB
    usage:   1.5 TiB used, 1.1 TiB / 2.6 TiB avail
    pgs:     35101/393459 objects misplaced (8.921%)
             94 active+clean
             34 active+remapped+backfill_wait
             1  active+remapped+backfilling

  io:
    client:   258 KiB/s rd, 49 MiB/s wr, 4 op/s rd, 267 op/s wr
    recovery: 32 MiB/s, 7 objects/s

And ceph osd status shows this.

ID  HOST                                                   USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE
 0  zorgo-production-worker-1.k8s.prod.zorgo.domain.root  27.9G   251G      0        0       0        0   exists,up
 1  zorgo-production-worker-0.k8s.prod.zorgo.domain.root   172G   107G      1      465k      0        0   exists,up
 2  zorgo-production-worker-3.k8s.prod.zorgo.domain.root   200G  78.9G     30     1263k      0        0   exists,up
 3  zorgo-production-worker-2.k8s.prod.zorgo.domain.root   175G   104G      1     1284k      0        0   exists,up
 4  zorgo-production-worker-4.k8s.prod.zorgo.domain.root   188G  90.9G     12      390k      1        0   exists,up
 5  zorgo-production-worker-5.k8s.prod.zorgo.domain.root   159G   119G      0     6552       0        0   exists,up
 6  zorgo-production-worker-6.k8s.prod.zorgo.domain.root   196G   123G      0      267k      0        0   exists,up
 7  zorgo-production-worker-7.k8s.prod.zorgo.domain.root   219G   100G      0        0       0        0   exists,up
 8  zorgo-production-worker-8.k8s.prod.zorgo.domain.root   199G   120G      1     7371       0        0   exists,up

I've tried unsetting these flags on every osd, not sure if I should reference them by osd.# or just # but I've tried both.

kubectl -n rook-ceph exec -i deploy/rook-ceph-tools -- ceph osd unset-group nodown,noin,noup,noout osd.0 osd.1 osd.2 osd.3 osd.4 osd.5 osd.6 osd.7 osd.8

How do I fix this issue?

[stemid]

Nevermind, now it's ok. I guess it took a while to resolve.