Skip to content
This repository has been archived by the owner on Aug 29, 2023. It is now read-only.

Implemented ESRGAN and GFPGAN upscaling #50

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

fAIseh00d
Copy link
Contributor

@fAIseh00d fAIseh00d commented Jun 23, 2023

Face upscaling implementation

app.py

Added an interface block that lists *.onnx files in the upscaler_models folder with radio buttons and a way to pass upscaler string to refacer.

refacer.py

reface function is modified so it creates a ESRGAN or GFPGAN objects if the upscaler string has corresponding values and makes self.upscale_en = True.
process_faces functions' code is modified so we can use face_swapper.get(frame, ... , paste_back=False) returning both extracted processed face and M affine matrix for scalar transform, that is then passed to the new paste_upscale.
paste_upscale function code is from INSwapper, heavily modified to remove unused code and fix jagged edge borders when the faces are partially on the edge of the frame.

gfpgan_onnx.py

Added a GFPGAN implementation based on xuanandsix's GFPGAN-onnxruntime-demo.

You can easily convert the models with the provided scripts. Converted GFPGANv1.4 model provides best identity compared to 1.3 and 1.2, which is crucial for such a small 128px face.

esrgan_onnx.py

Added a ESRGAN implementation based on my fork of Sg4Dylan's ESRGAN-ONNX.

In my tests the best quality and speed-wise model is converted 2x_DigitalFlim_SuperUltraCompact_nf24-nc8_289k_net_g, the original found in Model Database.
With that model the performance is almost the same as without upscaling.

It works with regular and compact ESRGAN models, it might even work with RealESRGAN, I haven't tested it.

how to use

Place downloaded *.onnx weights into upscaler_models folder, make sure GFPGAN models contain GFPGAN in the filename. Select model in the radio buttons interface block, None uses native INSwapper paste_back processing without upscale.

WARNING: big model weights warmup at first frames, so refacer kind of hangs for a moment. Then it works fine until you change models/restart.

notes

Tested on Windows with CPU/CUDA.

Keep in mind that only 128px face is upscaled, not the whole picture.

Both implementations are as simple as they get, they just convert a int8 numpy.array image to float32 tensor and process it (with 2 different normalizations).

ESRGAN-ONNX repo has a license conflict with Apache 2.0 and MIT being kind of incompatible, so the code in the PR written from scratch.

Personally I recommend using chaiNNer to convert ESRGAN models to *.onnx but it can be done with the code just using torch.
fp16 conversion don't have any performance advantages, at least on CUDA.

PS

Face masking still works so-so, big area around the face depending on an angle looks worse than the face itself and the rest of the image. I tried MODnet, but it has poor temporal consistency and sometimes misses face parts between frames.

If anyone can recommend any other matting/masking NN's that can be run on ONNX, I'll be grateful!

fAIseh00d and others added 2 commits June 23, 2023 21:24
Added ESRGAN and GFPGAN implementations
Copied and refactored INSwapper face paste affine transform code
Fixed visible jagged mask edges
---------
fAIseh00d <fAIseh00d@users.noreply.github.com>
@fAIseh00d fAIseh00d marked this pull request as ready for review June 25, 2023 13:09
@scf2k
Copy link

scf2k commented Jun 29, 2023

Result looks so much better when GFPGAN is used! Thanks for your PR!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants