How to generate high-quality images like the ones in Civitai?

[kissshhot]

sd-webui can be only used in windows, so I try diffusers instead.
But my results are not satisfactory. Here is my code:

from diffusers import StableDiffusionPipeline
import torch
from diffusers import DPMSolverMultistepScheduler

pipeline = StableDiffusionPipeline.from_single_file(
    "https://huggingface.co/gsdf/Counterfeit-V3.0/blob/main/Counterfeit-V3.0.safetensors",
    torch_dtype = torch.float16
)
pipeline.load_textual_inversion(
    '/data1/dyf/model/EasyNegative/',
    weight_name = 'EasyNegativeV2.safetensors',
    token='EasyNegative'
)
pipeline.load_lora_weights('/data1/dyf/lora/test/', weight_name = 'Dungeon_Marcille_SSA_V1.safetensors', adapter_name='test')
noise_schedule = DPMSolverMultistepScheduler(num_train_timesteps=1000, use_karras_sigmas=True)
pipeline.scheduler = noise_schedule
pipeline.to('cuda')
generator = torch.Generator('cpu').manual_seed(2665505224)

prompt = 'masterpiece,best quality,1girl,solo,marcillessa,red choker,<lora:Dungeon_Marcille_SSA_V1:1>,detailed and beautiful eyes,(cowboy shot:1.2), HAPPY,walking,jumping,(Turtleneck_sweater:1.4),(Leather_skirt:1.3)'

negative_prompt = '(worst quality, low quality, bad quality, normal quality:2), logo,text,blurry,low quality,bad anatomy,lowres,normal quality,monochrome,grayscale,worstquality,signature,watermark,cropped,bad proportions,out of focus,username,bad body,long body,(fat:1.2),long neck,deformed,mutated,mutation,ugly,disfigured,poorly drawn face,skin blemishes,skin spots,acnes,missing limb,malformed limbs,floating limbs,disconnected limbs,extra limb,extra arms,mutated hands,poorly drawn hands,malformed hands,mutated hands and fingers,bad hands,missing fingers,fused fingers,too many fingers,extra legs,bad feet,backlighting, bad-hands-5,badhandv4,negative_hand-neg,easynegative,FastNegativeV2,bad-picture-chill-75v,verybadimagenegative_v1.3,ng_deepnegative_v1_75t,fcHeatPortrait,an7,'

lora_scale = 1.0
images = pipeline(prompt, negative_prompt=negative_prompt, num_inference_steps=20, cross_attention_kwargs={"scale": lora_scale}, generator=generator, num_images_per_prompt = 4, clip_skip=2, guidance_scale=7).images
i = images[0]
i.save('/home/dyf/draw/output/test5.jpg')

and here is my result:

and I want to achieve this kind of effect:

I'm wondering:
1.Why is the quality of my image so poor? How can I improve it?
2.How can I use the same model, lora, sampler and other parameters as in Image 2?
3.If I have the following safetensor model files, how should I use them? Should I replace them in the unet folder?

Thanks a lot!

[asomoza]

Hi, the simpler and fastest answer is that the sd-webui, runs in linux, windows and mac since it's just python.

If you still want to use diffusers, there's a lot of problems in what you're doing, so I'll try to address them all:

First a 9GB SD 1.5 model is just wrong, use this one:

https://huggingface.co/gsdf/Counterfeit-V3.0/blob/main/Counterfeit-V3.0_fix_fp16.safetensors

Then, use the scheduler like this:

pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)
pipeline.scheduler.config.use_karras_sigmas = True

With that you can start getting some decent results, after that remove all the loading code from automatic1111, they just add noise to the prompt in diffusers, with this I mean:

<lora:Dungeon_Marcille_SSA_V1:1>
bad-hands-5,badhandv4,negative_hand-neg,FastNegativeV2,bad-picture-chill-75v,verybadimagenegative_v1.3,ng_deepnegative_v1_75t,fcHeatPortrait,an7

The ones in the negative prompt are all textual inversions so you will need to find the files and load them with the same method you're using for the EasyNegativeV2. There's lot of them so I'll just try with one, so leave the ones you loaded, for example EasyNegative as you set the token. I'm guessing you can load more than one but I don't use them so I really don't know.

You're also using a long negative prompt and prompt weighting, for that to work you'll need to load the lpw custom pipeline:

pipeline = StableDiffusionPipeline.from_single_file(
    "https://huggingface.co/gsdf/Counterfeit-V3.0/blob/main/Counterfeit-V3.0_fix_fp16.safetensors",
    torch_dtype = torch.float16,
    custom_pipeline="lpw_stable_diffusion",
)

Finally the image is a 1024x1024 one, so they probably upscaled it with the tile controlnet and used ADetailer, to do those in diffusers is a lot more harder than what you're doing so start simple first.

With this and a 768x768 output image, you should get something like this:

For the last part, your using from_single_file with that you can load just the safetensors models, you don't need the subdirectories, if you want to load another one, just replace the url with the new one or your local filepath, you can use the same models and loras than automatic1111.

For the schedulers, you can use this table to use the equivalent ones:

https://huggingface.co/docs/diffusers/api/schedulers/overview#schedulers

You won't get the same results as automatic1111 as they are completely different implementations.

[kissshhot]

It helps a lot! Thanks you very much!
I will try sd-webui later, could you please tell me how to change the size of output image to 768x768? I don't find that in Diffusers Docs.
and if I want to learn more about how to use diffusers or sd-webui on linux, what other materials can I learn from?

[asomoza]

your welcome, to set the size of the output add a width and height argument in the generation, in your code it would be like this:

images = pipeline(prompt, width=768, height=768, negative_prompt=negative_prompt, num_inference_steps=20, cross_attention_kwargs={"scale": lora_scale}, generator=generator, num_images_per_prompt = 4, clip_skip=2, guidance_scale=7).images

To learn how to use diffusers you can read the online documentation, you can start with the tutorial here. Diffusers works in a lot of models and tasks so you probably need to just focus on the stable diffusion parts. For example the height and width are here

For sd-webui you'll do better asking in their repository, I haven't used it in a long time, just at the beginning when stable diffusion 1.5 was released.

[kissshhot]

ok I get it.
Thank you again for your help!

[sayakpaul]

@kissshhot how were the results after the changes @asomoza suggested? :)

[asomoza]

there's a couple of issues I found with this though.

The first one is that when using the from_single_file doesn't load the community pipelines so the long prompt and prompt weighting didn't have any effect. This should be fixed with #6986.

The second one is that the lpw_stable_diffusion doesn't have implemented the clip_skip argument which is really important for anime checkpoints in SD 1.5, #6723 should fix this.

With both of them (I did a local temp fix) the result should be more similar to the ones in civit.ai

but would still need the tiled upscale and ADetailer to be comparable.

[sayakpaul]

Cool. We do support tiled decoding but tiled upscaling needs a separate upscaler no? What is ADetailer?

[asomoza]

Basically they're the fancy names for what they do under the hood:

Tiled Upscaling: Use the tile controlnet for upscaling (separate upscaler, usually 4x-ultrasharp) a tiled image while adding more details and without seams between them.
ADetailer: Use object detection (face, hands, person) and inpaint to fix and add details to them automatically after the generation

With these two, the results are of the same quality of what people get in Automatic1111.

[sayakpaul]

Hmm looks interesting. @yiyixuxu do you think we should integrate it or maybe better to have them external to diffusers to keep the library lightweight?

[yiyixuxu]

probably community pipelines?

cc @vladmandic here too

would these(Tiled Upscaling and ADetailer) benefit sd.next as well?

[yiyixuxu]

cc @stevhliu here
maybe we can put together a guide too

thanks @asomoza you're a rock star!!

[vladmandic]

my $0.02

tiled-diffusion was popular, but less so nowadays and standard hires workflow is more common given that gpus have sufficient vram.
(hires workflow is generate image at close to models native resolution, upscale it and use it as input to img2img to avoid model halucinations). on the other hand, tiled diffusion is useful only if you dont have enough vram to start with - and then its really slow since it needs to generate with significant overlaps to avoid seams.

adetailer is very popular (and its exactly what @asomoza said - it runs detection of a specific object type (most commonly face), crops the image and runs its through img2img pipeline at higher resolution than cropped and then resizes back and pastes over original results). i'd love to have it as a more native item, but so far have been relying on adetailer extension given everything else on my plate. big question is also which models to use for detection if desire is to integrate it more closely - by default it uses mediapipe which has its own pipeline and its not the best solution, especially given common installation issues. i'd prefer to rely on torch-based model so it can be run using existing engine.

bigger question is how much should be done in the underlying library vs app vs app extensions
diffusers is a library. it can be used by user directly if he's a dev, but its not an app. and task of writing an app like sdnext or a1111 is massive (regardless if its using diffusers or ldm or whatever as starting point). even then, not everything can be done even in the app natively, that's why there is a large community developing app extensions.

[crapthings]

@asomoza load multiple textual inversion cause image very dark, looks like afternoon or night
what is wrong? by the way i'm using it with multiple controlnet too

[asomoza]

I tried with a couple and they worked, if they can be downloaded it would help if you share the links to download them. Also try them without controlnet, if they work without it, then I'll need a reproducible code snippet to be able to test and see what the problem is.

[crapthings]

okay, let me make a notebook