-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
training example for instruct pix2pix doesn't zero out embeds #7920
Comments
Thanks for bringing this up. Possible for you to show a comparison between what happens when you zero out like the way you mentioned compared to the existing approach?
The SD IP2P pipeline uses diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.py Line 558 in ec9e881
However, it makes use to diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.py Line 869 in ec9e881
|
the base model was trained using it, so i figured aligning with the base model's training and inference has better results. from my own tests, i can now reduce the step count required when running the default config on the SDXL pipelines, eg. force zeroes is set to True I also have much better learning. this model started from 1000 steps of tuning later: the base model was trained using |
see the base SDXL pipeline: # get unconditional embeddings for classifier free guidance
zero_out_negative_prompt = negative_prompt is None and self.config.force_zeros_for_empty_prompt
if do_classifier_free_guidance and negative_prompt_embeds is None and zero_out_negative_prompt:
negative_prompt_embeds = torch.zeros_like(prompt_embeds)
negative_pooled_prompt_embeds = torch.zeros_like(pooled_prompt_embeds) and the config: {
"_class_name": "StableDiffusionXLPipeline",
"_diffusers_version": "0.19.0.dev0",
"force_zeros_for_empty_prompt": true
} |
Ah makes a ton of sense. Do you want to take a step at opening a PR to fix this? |
can i also open the pull request for all of the other training examples, to add general dropout capabilities to them? |
We can open that up for the community. This way everyone gets to participate. |
like the ticket for updating the fp16 error? #6231 |
Describe the bug
When running inference on SDXL, the config specifies to zero out the embedding when the prompt is empty.
Reproduction
this could likely be replaced with a probabilistic call to
torch.zeros_like()
inside the training loop instead.I've checked the values of the embeds, and classifier-free guidance at inference time definitely makes use of the zero embed and not just
""
, which end up producing very different results.other models though like deepfloyd just use
""
from eg. T5 and behave rather differently.Logs
No response
System Info
N/A
Who can help?
@sayakpaul
The text was updated successfully, but these errors were encountered: