[Feature Request] Support for SDXS-512, allowing for real-time image generation on the CPU (~0.6 seconds per image)

[JohnAlcatraz]

It would be great if support for the SDXS-512 model could be added: https://github.com/IDKiro/sdxs

Especially for quickly generating images on the CPU, this is a major new development, so it would fit this project really well. https://github.com/rupeshs/fastsdcpu already supports it (requires Python though and really isn't lightweight and dependency-free like stable-diffusion.cpp).

I benchmarked SDXS-512 in FastSDCPU on my CPU (3950X):

FastSDCPU Regular:
RAM Usage: 700 MB
Image Generation Time (Latency): 2 seconds

FastSDCPU OpenVINO:
RAM Usage: 3.6 GB
Image Generation Time (Latency): 0.6 seconds

So it's really very fast, and even super memory efficient (when not using OpenVINO). And it even looks much better than the best reasonably fast results I can get with stablediffusion.cpp, which is using Dreamshaper LCM with ~5 steps, 2 GB RAM usage and 19 seconds generation time. SDXS-512 looks better, is 20 times faster and uses only 1/3 the RAM - and that's with a Python implementation.

[Amin456789]

indeed, if its gonna faster than sdxl turbo 512x512 with 1step then it is gonna be great to have it

have u tried it already? maybe its gonna work, we got sdxl lighitng and pony realsim to work without sd cpp get a new update. maybe this gonna work too

[Amin456789]

@IDKiro could u please take a look at this, we can use this for koboldcpp for chat and image generation, thank u

[IDKiro]

@IDKiro could u please take a look at this, we can use this for koboldcpp for chat and image generation, thank u

Yes, SDXS is much faster than SD Turbo, and I recommend using the SDXS-512-DreamShaper we put out, which is compatible with original SD1.5 VAE and Tiny VAE.
The SDXS-512-DreamShaper is based on SD 1.5, but with changes in the structure of the UNet, and we have marked the changes in the diffusers configuration file:
{
"block_out_channels": [
320,
640,
1280
],
"down_block_types": [
"DownBlock2D",
"CrossAttnDownBlock2D",
"CrossAttnDownBlock2D"
],
"layers_per_block": 1,
"up_block_types": [
"CrossAttnUpBlock2D",
"CrossAttnUpBlock2D",
"UpBlock2D"
],
}