Outpainting with Diffusers

[asomoza]

Hi, I have made a 3 part series about outpainting with Diffusers, each one with a different method.

Outpainting I - Controlnet version

Outpainting using a controlnet trained for this. This method does change the original image but has the advantage that the ourpaining its more creative and seamless.

Outpainting II - Differential Diffusion

This method uses differential diffusion which allows us to use a soft inpainting mask that generates seamless outpainting, as the first method I can change the original image but not as much.

Outpainting III - Inpaint Model

The final method uses an inpainting model but it will preserve the original image so it's more suitable for professional work.

[tolgacangoz]

[asomoza]

fixed, thanks.

[crapthings]

control_image = Image.open(BytesIO(response.content))
new_controlnet_image = Image.new("RGBA", control_image.size, "WHITE")
new_controlnet_image.alpha_composite(control_image)

why this dones't work?

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[11], line 17
     14 b = midas(input_image)
     16 new_controlnet_image = Image.new('RGB', input_image.size, 'WHITE')
---> 17 new_controlnet_image.alpha_composite(ccc)
     18 # new_controlnet_image = Image.composite(input_image, new_controlnet_image, input_image)
     20 display(new_controlnet_image)

File [~/.local/lib/python3.10/site-packages/PIL/Image.py:1790](http://192.168.50.251:8888/lab/tree/workspace/ai-pipe/ai-pipe-replace-model-new/.local/lib/python3.10/site-packages/PIL/Image.py#line=1789), in Image.alpha_composite(self, im, dest, source)
   1787 else:
   1788     background = self.crop(box)
-> 1790 result = alpha_composite(background, overlay)
   1791 self.paste(result, box)

File [~/.local/lib/python3.10/site-packages/PIL/Image.py:3328](http://192.168.50.251:8888/lab/tree/workspace/ai-pipe/ai-pipe-replace-model-new/.local/lib/python3.10/site-packages/PIL/Image.py#line=3327), in alpha_composite(im1, im2)
   3326 im1.load()
   3327 im2.load()
-> 3328 return im1._new(core.alpha_composite(im1.im, im2.im))

ValueError: image has wrong mode

[asomoza]

When you're doing alpha composite you need both images to have an alpha channel. Probably the image you're loading as a control_image is a RGB image instead of a RGBA

[juzstu]

I find that the BrushNet also performs well in outpainting.

[asomoza]

Thanks, I have it under my radar, I'm waiting for the SDXL version but it seems that's going to take more time.

[juzstu]

@asomoza For your Outpainting III - Inpaint Model, I use lora to finetune the RealVisXL_V4.0, but the image quality was really bad, if not use the lora weights, the image quality turn to normal, can you tell me what case this and how can I use lora weights correctly to generate a normal quality image?

[asomoza]

That is a step by step guide, I provide a full script at the end but it's just so people can learn and follow it.

If you're just going to use it and change something, you should divide it into two parts, one for generating the temp background and the second part to actually generate the background with an inpainting model. Also you'll need to remove the fixed seeds from the code.

I'll need a little more information on what you're trying to do, are you loading the LoRA in the first step? the second? both? For what are you using the LoRA? the LoRA works ok if you just use it with a normal generation?

For example, if I just want to change the style of the generated background, I can just load the pixel art LoRA on the second step, and change the prompt for that one with the trigger for the LoRA.

source image	bg removed

temp background	final background	pixel art backgound

Hope the images explain the process a little better, but for me to be able to help you, I'll need more information on what you're trying to do.

[juzstu]

Thanks for your replay. I use the my own data to lora finetune the RealVisXL_V4.0 and use it in the first step, if I drop the zoe-depth model from the controlnets, the image will be normal. As you mentioned that the inpaint model likes to add details, but if I use zoe-depth or canny/depth mdel, the background only has pure color and no or just little details.

[asomoza]

In my last example I changed the source image, the prompts, the seeds and added a LoRA and it worked just fine, the controlnet works as intended even if you change those.

So the problem lies in one or a couple of what I mentioned, I'm guessing you're not sharing those because you can't, so I really can't help you anymore with the information you can give me.

My last suggestion is that you can try different settings for the depth controlnet, try lowering the controlnet_conditioning_scale or the control_guidance_end, maybe your LoRA needs more freedom to work.

[juzstu]

Thank you for your suggestion, I will give it a try.