[add pipeline]: A new Pixart-based inpainting pipeline. #7929

eightmusic · 2024-05-13T06:36:56Z

We add a pixart model-based inpaint method that works similarly to pipeline_stable_diffusion_inpaint. @sayakpaul @yiyixuxu @DN6

sayakpaul · 2024-05-13T09:37:34Z

Could we see some results here to decide if it's worth adding as a core pipeline?

Cc: @lawrence-cj

yiyixuxu · 2024-05-13T19:34:29Z

cc @asomoza here too!

eightmusic · 2024-05-17T06:53:00Z

Could we see some results here to decide if it's worth adding as a core pipeline?

Cc: @lawrence-cj

prompt='Face of a yellow cat, high resolution, sitting on a park bench'

prompt=''

This is an image of the effects of a different prompt, which the pipeline has merged with the official PixArt repository.@sayakpaul @yiyixuxu @DN6 @lawrence-cj

sayakpaul · 2024-05-17T07:28:36Z

@lawrence-cj could you review this too?

HuggingFaceDocBuilderDev · 2024-05-17T07:33:34Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

sayakpaul · 2024-05-17T07:38:00Z

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_inpaint.py

@@ -0,0 +1,1097 @@
+# Copyright 2023 PixArt-Alpha Authors and The HuggingFace Team. All rights reserved.


Suggested change

# Copyright 2023 PixArt-Alpha Authors and The HuggingFace Team. All rights reserved.

# Copyright 2024 PixArt-Alpha Authors and The HuggingFace Team. All rights reserved.

sayakpaul · 2024-05-17T07:38:33Z

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_inpaint.py

+        >>> img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
+        >>> mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"
+
+        >>> init_image = download_image(img_url).resize((512, 512))


Could replace this with diffusers.utils.load_image(). Less lines of code. WDYT?

sayakpaul · 2024-05-17T07:39:14Z

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_inpaint.py

+        ```
+"""
+
+ASPECT_RATIO_1024_BIN = {


Do we wanna move these constants to the init of pixart as these are now access by two pipelines? WDYT?

Cc: @lawrence-cj WDYT?

It sounds reasonable for me.

sayakpaul · 2024-05-17T07:39:58Z

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_inpaint.py

+    def encode_prompt(
+            self,
+            prompt: Union[str, List[str]],
+            do_classifier_free_guidance: bool = True,
+            negative_prompt: str = "",
+            num_images_per_prompt: int = 1,
+            device: Optional[torch.device] = None,
+            prompt_embeds: Optional[torch.FloatTensor] = None,
+            negative_prompt_embeds: Optional[torch.FloatTensor] = None,
+            prompt_attention_mask: Optional[torch.FloatTensor] = None,
+            negative_prompt_attention_mask: Optional[torch.FloatTensor] = None,
+            clean_caption: bool = False,
+            **kwargs,
+    ):


We should use the same method from the PixArtAlpha implementation and use a # Copied from ... statement here.

sayakpaul · 2024-05-17T07:41:20Z

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_inpaint.py

+            extra_step_kwargs["generator"] = generator
+        return extra_step_kwargs
+
+    def check_inputs(


If this copied from the text-to-image pipeline implementation, let's use the # Copied from ... statement here. Applies here and elsewhere.

But here, I think the function should also validate the input image and the mask image. Here's a reference:

diffusers/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py

Line 778 in 6c60e43

def check_inputs(

sayakpaul · 2024-05-17T07:41:43Z

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_inpaint.py

+    def classify_height_width_bin(height: int, width: int, ratios: dict) -> Tuple[int, int]:
+        """Returns binned height and width."""
+        ar = float(height / width)
+        closest_ratio = min(ratios.keys(), key=lambda ratio: abs(float(ratio) - ar))
+        default_hw = ratios[closest_ratio]
+        return int(default_hw[0]), int(default_hw[1])
+
+    @staticmethod
+    def resize_and_crop_tensor(samples: torch.Tensor, new_width: int, new_height: int) -> torch.Tensor:


# Copied from ... statement missing.

Recommits have been made in the appropriate format.@sayakpaul

sayakpaul

The pipeline implementation looks very nice to me, thank you so much!

I think what's pending for this PR to get merged is the following:

Docs
Test

… the code

yiyixuxu

thanks for adding this pipeline!
+1 on @sayakpaul 's comments about doc and tests

yiyixuxu · 2024-05-20T16:22:22Z

src/diffusers/pipelines/pixart_alpha/pipeline_pixart_inpaint.py

+                If `return_dict` is `True`, [`~pipelines.ImagePipelineOutput`] is returned, otherwise a `tuple` is
+                returned where the first element is a list with the generated images
+        """
+        if "mask_feature" in kwargs:


we don't need to deprecate for new pipeline

sayakpaul · 2024-05-21T03:25:46Z

@eightmusic let's get the comments resolved and we would be happy to include in the new release we're cooking.

Syncing changes from the main branch

eightmusic · 2024-05-21T07:09:51Z

@eightmusic let's get the comments resolved and we would be happy to include in the new release we're cooking.
ruff check examples scripts src tests utils benchmarks setup.py
src/diffusers/pipelines/pixart_alpha/pipeline_pixart_inpaint.py:15:1: I001 [] Import block is un-sorted or un-formatted
src/diffusers/pipelines/pixart_alpha/pipeline_pixart_inpaint.py:23:31: F401 [] torch.nn.functional imported but unused
src/diffusers/pipelines/pixart_alpha/pipeline_pixart_inpaint.py:454:1: W293 [] Blank line contains whitespace
src/diffusers/pipelines/pixart_alpha/pipeline_pixart_inpaint.py:509:1: W293 [] Blank line contains whitespace
src/diffusers/pipelines/pixart_alpha/pipeline_pixart_inpaint.py:1092:49: W292 [] No newline at end of file
Found 5 errors.
[] 5 fixable with the --fix option.
make: *** [Makefile:43: quality] Error 1
These problems have been solved.@sayakpaul

sayakpaul · 2024-05-21T09:40:20Z

Thank you!

There are still open comments left to be addressed. Additionally, we need to have docs and tests for this to get merged.

eightmusic · 2024-05-21T09:47:44Z

Thank you!

There are still open comments left to be addressed. Additionally, we need to have docs and tests for this to get merged.

The previous comments should be resolved, is there anything else I need to do?

sayakpaul · 2024-05-21T09:54:54Z

Well, the tests are failing. We need to get them sorted. And then there are comments still open, e.g.: #7929 (review), #7929 (comment).

And then we need to add tests for the pipeline and documentation. LMK if there's anything unclear.

eightmusic · 2024-05-21T10:09:01Z

#7929 (review), #7929 (comment) problem has just been solved.'We need to get them sorted.' What do I need to do here. Are there any references to test and docs? I don't know what this is.@sayakpaul

sayakpaul · 2024-05-21T10:13:43Z

Are there any references to test and docs?

Here's how an inpainting test suite should be devised: https://github.com/huggingface/diffusers/blob/main/tests/pipelines/stable_diffusion_2/test_stable_diffusion_inpaint.py.
Here's how a pipeline doc should be added: https://github.com/huggingface/diffusers/blob/main/docs/source/en/api/pipelines/pixart.md.

Quality tests can be fixed by running make style && make quality. Refer to the contribution guidelines for more details.

lawrence-cj · 2024-05-22T06:59:13Z

Sry to disturb you, but I found I can't push a commit to this branch, any suggestions? @sayakpaul

sayakpaul · 2024-05-22T07:00:44Z

I think you will need for submit a PR to https://github.com/eightmusic/diffusers/. Perhaps @eightmusic could add you as a collaborator to the repo?

lawrence-cj · 2024-05-22T07:03:03Z

Agree. I can help adding the make style and test part. Is it possible to add me as a collaborator to your forked diffusers repo? @eightmusic

eightmusic · 2024-05-22T07:07:48Z

Agree. I can help adding the make style and test part. Is it possible to add me as a collaborator to your forked diffusers repo? @eightmusic

of course.

[add pipeline]: A new Pixart-based inpainting pipeline.

2c3453e

Merge branch 'main' into inpaint

c20b074

sayakpaul requested a review from yiyixuxu May 17, 2024 07:28

sayakpaul reviewed May 17, 2024

View reviewed changes

Add appropriate descriptions and modify the "check_inputs" section of…

8360483

… the code

yiyixuxu reviewed May 20, 2024

View reviewed changes

sayakpaul and others added 3 commits May 21, 2024 08:55

Merge branch 'main' into inpaint

653c8f7

fix formatting issues

c515e26

Merge remote-tracking branch 'origin/inpaint' into inpaint

b8885b4

Syncing changes from the main branch

Merge branch 'main' into inpaint

13f89a8

remove deprecate

1860c27

fix bug

75a8e47

lawrence-cj and others added 4 commits May 22, 2024 15:13

make style

2a83c20

Merge remote-tracking branch 'refs/remotes/origin/main' into inpaint

93d4fb8

Add PixArtAlphaInpaintPipeline to __init__

55e877d

add docs

d0b2979

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[add pipeline]: A new Pixart-based inpainting pipeline. #7929

[add pipeline]: A new Pixart-based inpainting pipeline. #7929

eightmusic commented May 13, 2024

sayakpaul commented May 13, 2024

yiyixuxu commented May 13, 2024

eightmusic commented May 17, 2024

sayakpaul commented May 17, 2024

HuggingFaceDocBuilderDev commented May 17, 2024

sayakpaul May 17, 2024

sayakpaul May 17, 2024

sayakpaul May 17, 2024

lawrence-cj May 20, 2024

sayakpaul May 17, 2024

sayakpaul May 17, 2024

sayakpaul May 17, 2024

eightmusic May 17, 2024

sayakpaul left a comment

yiyixuxu left a comment

yiyixuxu May 20, 2024

sayakpaul commented May 21, 2024

eightmusic commented May 21, 2024

sayakpaul commented May 21, 2024 •

edited

eightmusic commented May 21, 2024

sayakpaul commented May 21, 2024

eightmusic commented May 21, 2024

sayakpaul commented May 21, 2024

lawrence-cj commented May 22, 2024

sayakpaul commented May 22, 2024

lawrence-cj commented May 22, 2024 •

edited

eightmusic commented May 22, 2024

		@@ -0,0 +1,1097 @@
		# Copyright 2023 PixArt-Alpha Authors and The HuggingFace Team. All rights reserved.

	# Copyright 2023 PixArt-Alpha Authors and The HuggingFace Team. All rights reserved.
	# Copyright 2024 PixArt-Alpha Authors and The HuggingFace Team. All rights reserved.

[add pipeline]: A new Pixart-based inpainting pipeline. #7929

Are you sure you want to change the base?

[add pipeline]: A new Pixart-based inpainting pipeline. #7929

Conversation

eightmusic commented May 13, 2024

sayakpaul commented May 13, 2024

yiyixuxu commented May 13, 2024

eightmusic commented May 17, 2024

sayakpaul commented May 17, 2024

HuggingFaceDocBuilderDev commented May 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayakpaul left a comment

Choose a reason for hiding this comment

yiyixuxu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayakpaul commented May 21, 2024

eightmusic commented May 21, 2024

sayakpaul commented May 21, 2024 • edited

eightmusic commented May 21, 2024

sayakpaul commented May 21, 2024

eightmusic commented May 21, 2024

sayakpaul commented May 21, 2024

lawrence-cj commented May 22, 2024

sayakpaul commented May 22, 2024

lawrence-cj commented May 22, 2024 • edited

eightmusic commented May 22, 2024

sayakpaul commented May 21, 2024 •

edited

lawrence-cj commented May 22, 2024 •

edited