Tweak lossy modular #3575
Open
Tweak lossy modular #3575
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jonsneyers
force-pushed
the
tweak_lossy_modular
branch
3 times, most recently
from
33b1660
to
bd8aac6
Compare
jonsneyers
force-pushed
the
tweak_lossy_modular
branch
from
bd8aac6
to
77796b1
Compare
jonsneyers
force-pushed
the
tweak_lossy_modular
branch
from
77796b1
to
db9382d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(builds on top of #3563)
Better align lossy modular quality with lossy vardct quality. Also tweaking the quantization choices in lossy modular, so it actually applies B quantization to B instead of using X quantization for both X and B (which was what it was effectively doing). Tweaking the allocation of bits between luma (Y) and chroma (X,B). Also slightly improves generation loss.
This alignment was needed because lossy modular and lossy vardct were producing quite diverging results for the same distance setting.
Before
These are lossy modular results for a range of distance settings:
These are VarDCT results (after #3563) for those same distance settings:
The alignment is not very good: at d > 1, lossy modular produces significantly larger files (and better quality) than vardct; at d < 1 it is the other way around.
After
VarDCT doesn't change, so see above.
Modular after this PR:
Alignment is better this way.
To get a fair comparison of old modular vs tweaked modular, here is old modular at similar bpp (slightly higher) as the tweaked modular:
It's not a big difference if you compare it this way. It depends a bit on the metric; generally the tweaked modular does get slightly better scores at slightly lower bpp, but it's not very clear.
Generation loss
Now if we take those same settings and compare old vs tweaked after 100 generations, the difference does become a bit clearer:
Before
Generation loss testing with 100 intermediate generations
After
Generation loss testing with 100 intermediate generations
So the tweaked modular overall produces better quality after 100 generations, at slightly lower bpp.
For comparison, this is VarDCT after 100 generations (and after #3563, which improves generation loss at lower distances):
Generation loss testing with 100 intermediate generations
(ignore the aggregate for ssimulacra2, we need to fix that when the scores can get negative like they can)
Before #3563 (i.e. current git head), it was even worse, and it seems like d < 1 even produces worse artifacts than d1 after generation loss — presumably because encoder gaborish compensates for d1 artifacts but overcompensates for it when using a lower distance:
Generation loss testing with 100 intermediate generations
Conclusion
With these changes, the alignment between lossy modular and lossy vardct improves, making both modes behave more similarly when using the same distance setting.
For single-generation encoding (which is the bulk of the use cases), vardct mode is still clearly preferable in all aspects (compression density, encode speed, decode speed). However, for multi-generation encoding, lossy modular does have benefits: it suffers significantly less from generation loss. Whether or not that is a good trade-off that offsets the worse single-generation compression density and slower encode/decode, depends on the use case (I assume usually it doesn't).