New features and feature requests #87
Replies: 14 comments 13 replies
-
not sure what you mean by that exactly, but sounds interesting ;-) can you elaborate further? excited about further config-options for blending etc., too! well, you know what's on @anxiousottergames ' and my wishlist.. ;-p but like I already said: you focus on what you deem necessary, first. it's coming along nicely :-) |
Beta Was this translation helpful? Give feedback.
-
With Deepface Lab you can swap a whole head but this needs a lot of training. Having the possibility to transfer at least hairstyles would be great I think...I toyed around with this but it's unusable slow. |
Beta Was this translation helpful? Give feedback.
-
|
bummer.. maybe something faster will show up eventually. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good to me. I'm not sure if you had them listed in order of importance but I personally feel like even the order of the planned features listed is perfect. |
Beta Was this translation helpful? Give feedback.
-
|
With Roop development now shut down, this is the only place where we can expect improvements on this nice tool. After a lot of tests, I figured out (at least within my test results) that the biggest flip side is the eye movement. It is totally missing in a video swap. The model seems to be looking straight forward, irrespective of the actions. Hence, the model looks like visually impaired or blind. This is not noticeable in a still image swap because it is a still image, there is no action. But in a video, this lack of eye movement is very easily noticeable. You can tell within 4-5 seconds that the video is fake. Above all of the other enhancements discussed, I think we first need to focus on how to copy the eye-movements of the target to the output while we swap the face from the source. Basically, the face will come from the source, we need to detect the eyeballs, copy the eyeball direction from the target (not using the source eyeball directions). That will add life to our end product. This technology is already there because Adobe Photoshop can do this. We can change eye direction in a photo using Photoshop. We have to do that same thing with the output image after detecting the direction from the target frames. But I am not sure if there is an open source project already available for this. A close match would be Gaze Correction, but it has limited features. It may be used as a start point. |
Beta Was this translation helpful? Give feedback.
-
|
That's why I linked https://github.com/ai-forever/ghost which especially cared about gaze and emotions. From my limited testing it did this better than the rest but the face mask was lacking or rather looked too artificial. |
Beta Was this translation helpful? Give feedback.
-
|
What I am doing right now is, I am using multiple source images having various angles. This is not difficult to do, if we have a source video and we extract several source frames before the swapper is called from main. Then in the swapper, I am selecting the best match (from the source image set) based upon the face angle of the target frame. This is giving a better result, at least there are eye movements. But these movements are not fully realistic, because they are not entirely driven by the target. Some half-way solution until we discover a better approach. I will look into Ghost and see if we can borrow just the eye tracking part from there and add a new processor between the swapper and the enhancer to rectify the gaze. Thanks for the leads! |
Beta Was this translation helpful? Give feedback.
-
|
Can you share the code for this? Would love to try |
Beta Was this translation helpful? Give feedback.
-
This is actually quite a good idea, perhaps even better than the blending of several images. |
Beta Was this translation helpful? Give feedback.
-
|
can we get a fps changer (probably doable with ffmpeg even without reencoding) this is useful for some high resolution videos that are 60fps and because of this takes a lot of time but can be 30fps cutting processing time in half , maybe processing every second frame instead of every frame ? Also like in stable diffusion extension can we select faces by number they appear on screen ? (1 and 3 for example out of 4 faces) Both of these are especially useful for 180vr videos where a persons face appear two times but since they are in different angles app doesnt recognize both as the same almost always. This is not a problem if only one person is in the video just select all faces BUT if there are more than one person on screen we either need to change all faces or just one persons face on one side of the video. |
Beta Was this translation helpful? Give feedback.
-
|
I tested Fast Segment-Anything yesterday and am quite disappointed. It's faster than the current Clip2Seg but surprisingly very bad when using low-resolution inputs. If it can identify objects, the resulting masks are way better though. I don't know if it's worth the hassle. |
Beta Was this translation helpful? Give feedback.
-
|
Suggestion on the Side: What about an Upload Source for an Audio File (mp3) for "directly" a new Lip Sync? -> Overwrits the Lip Sync of the Target Video and Change it with the "Manuellextra located Audio File. :-) |
Beta Was this translation helpful? Give feedback.
-
|
Hm...uh...I don't know. With an audiofile of the exact same size this is very easy of course but I don't want to open a can of worms, because of different audio formats, weird sizes, special ffmpeg handling. I'd rather leaves this to other great, specialized tools like this |
Beta Was this translation helpful? Give feedback.
-
|
but isnt it lossless-cut only a editing programm? you can limit the usage data of the audio files in roop. f.e. so only mp3 is allowed, or something. sadtalker is really nice, but only can lipsync to a single file/frame. i made my cosuin a birthday present with his face on a famous movie scene, where the actor sing a song. i used another song and cut it together in after effects. but the lip sync was - of course wrong (on the old/original audiofile. so here it would be perfekt if i could adjust the lipsync with an audio file of my desire |
Beta Was this translation helpful? Give feedback.
-
Yes and adding a custom audio track is editing. From its description: The idea with your cousin is really neat but I won't do it. |
Beta Was this translation helpful? Give feedback.
-
|
Controlnet?Sent from my iPhoneOn Aug 21, 2023, at 6:24 AM, JeetGuhaThakurta ***@***.***> wrote:
With Roop development now shut down, this is the only place where we can expect improvements on this nice tool. After a lot of tests, I figured out (at least within my test results) that the biggest flip side is the eye movement. It is totally missing in a video swap. The model seems to be looking straight forward, irrespective of the actions. Hence, the model looks like visually impaired or blind.
This is not noticeable in a still image swap because it is a still image, there is no action. But in a video, this lack of eye movement is very easily noticeable. You can tell within 4-5 seconds that the video is fake. Above all of the other enhancements discussed, I think we first need to focus on how to copy the eye-movements of the target to the output while we swap the face from the source.
Basically, the face will come from the source, we need to detect the eyeballs, copy the eyeball direction from the target (not using the source eyeball directions). That will add life to our end product. This technology is already there because Adobe Photoshop can do this. We can change eye direction in a photo using Photoshop. We have to do that same thing with the output image after detecting the direction from the target frames. But I am not sure if there is an open source project already available for this. A close match would be Gaze Correction, but it has limited features. It may be used as a start point.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
I don't know how or why ControlNet should be added to a face swapper? This is no diffuser and no inpainting so far... |
Beta Was this translation helpful? Give feedback.
-
I'm searching for the possibility to erode and blur the face mask but on the last roop-unleashed (3.3.4) I can't find the options, is that possible? I'm asking that 'cause i'm still obtain a slight ghost-box around my swapped face and i want to improve the results blurring the box a bit. |
Beta Was this translation helpful? Give feedback.
-
|
Since I had a potato computer, I integrated gpen 256 thinking it was better than nothing. But I couldn't get the Original/Enhanced image blend ratio part to work correctly. It only works between 0.90 and 1. When it goes below that, it adds strange blurs and distorts the image. what would be the reason ? |
Beta Was this translation helpful? Give feedback.
-
|
did you just replace the 512 model with the 256 version, or have you made changes to the code? my guess would be, that (at the very least) you have to adjust Enhance_GPEN.py at line 23 (model_path = resolve_relative_path('../models/GPEN-BFR-512.onnx')) and 34 (temp_frame = cv2.resize(temp_frame, (512, 512), cv2.INTER_CUBIC)). can't say if that's the only thing you'd have to adjust (probably not 😆) - you'd have to ask @C0untFloyd (still on vacation) |
Beta Was this translation helpful? Give feedback.
-
|
@C0untFloyd quick question. I noticed the TEMP folder gets really chunky after extended use and keeps facesets duplicated from the UI. Can i periodically delete the contents of the temp folder? |
Beta Was this translation helpful? Give feedback.
-
|
yes you can, there's even a button for it in the settings tab. |
Beta Was this translation helpful? Give feedback.
-
|
Transfered here from the discussion started by @aripsam Since this version is pretty stable and has been running well for a while now, I'm just wondering if there are any plans to add any more features? Anything in the pipeline? Some features I can think of -
|
Beta Was this translation helpful? Give feedback.
-
|
Well, it's not dead yet but admittedly development slowed down a lot because of the holidays, a lot of real life work and I still did not fully recover from my leg injury. I merged the PR for auto rotation into the dev-branch and would like to release this into main but before that I would like to improve this to rotate every face instead of the whole image. This isn't quickly done, as the PR is huge and I want to get it right. I also integrated a VR mode for swapping faces in VR images/movies but this too needs some more work to cope with fisheye distortion etc.
About your requested features, the first 3 were probably inspired by Rope? 😄 |
Beta Was this translation helpful? Give feedback.
-
|
i also have a suggestion, maybe its caused by its limitation? |
Beta Was this translation helpful? Give feedback.
-
|
Insightface Swapper Resolution. Please see the 2nd issue to ever grace this repo 😄 >>--> #2 |
Beta Was this translation helpful? Give feedback.
-
|
Btw. I forked a new promising hairstyle transfer repo today, which would be a great addition to roop unleashed once it is working: I can't get it to work so far though, not in Colab and not on my machine. Perhaps one of you can enlighten me, the problem is the dynamic compilation of python modules which I can't seem to get right. Also there is a lot of hopefully unnecessary Jupyter stuff in it, this would have to be removed for our own local needs. |
Beta Was this translation helpful? Give feedback.
-
Here I would like to collect and show planned features for roop unleashed.
My wishlist so far would be:
Beta Was this translation helpful? Give feedback.
All reactions