main should be left alone so things we forget have somewhere to live.

I'm not thrilled about adding a new scheduling group but it's the right thing to do.

Better is to have nested groups (and so commitlog/data and commitlog/schema).

nested sched groups: scylladb/seastar#1070

Also, what other tasks are running on main?

First, extremely important start, stop and config-reread code all run in main sg :)
Second, recent logs contain current sched group "name" next to shard id, so it can be found out by grepping it

@kostja please decide who should work on this -- storage group or your group?

@kbr-scylla depends on how you suggest to fix it. Nested scheduling groups?

Not necessarily, if it's too much effort we could simply have a separate normal scheduling group.

@kostja please decide who should work on this -- storage group or your group?

@kostja ping

I think this should belong to Storage.

So this issue actually breaks down into 3 or 4 related items.

Let me first write what the goal of this issue is: we want to make sure that there is no priority inversion between the data path and the schema path. E.g. when a node needs to make a read barrier to fetch the latest schema, and this barrier needs to write schema mutations to the schema commit log, these writes are not queued behind data path writes to data commit log and compaction. At the end of the day it manifests with latency spikes in clusters which alter tables under heavy load (todo: we need a link to the original issue here).

In order to fix that we need to do multiple things:

1. change the scheduling group for the commit log; this seems to be trivial, and can be done when creating the commit log in system_tables.cc
2. We need to make sure the schema commit log is using a differetn IO priority class than the rest of the writes. This is not implemented, neither in the commit log API nor at the instantiation place.
3. we need to retest to see if it helps.

At the end of the day it manifests with latency spikes in clusters which alter tables under heavy load (todo: we need a link to the original issue here).

#15622

The issue is fixed.

Schema commit log is already using a separate scheduling group from regular commitlog -- the main scheduling group, but as @avikivity said, "main should be left alone so things we forget have somewhere to live."

The above issue was caused by a priority inversion as you described, but the problem was not commitlog as initially suspected, it was using the user data memory pool for Raft tables -- so schema writes were waiting for user memtable flushes.

I think there is no longer a concept of IO priority classes, they were merged with scheduling groups (?)

There really is nothing else to be done than creating a separate scheduling group for schema commitlog, I think.

I think there is no longer a concept of IO priority classes, they were merged with scheduling groups (?)

This is correct.

@kbr-scylla if the issue is fixed why don't we close this one?

Sorry for confusion @eliransin I meant that #15622 is fixed (which was the cause of latency spikes that Kostja mentioned)

The current issue is not fixed, schema commitlog is in main group while (I believe) it should have its own

Will it make to 6.0?

See also #1926