I can generate 512x512 images in 95ms per image.(4090)

[aifartist]

With the code for LCM dropping today I've already started thinking about audio feedback realtime video. I just found your thing here right before I need to sleep so I need to see if I can leverage both LCM and what you've already done.

100%|███████████| 4/4 [00:00<00:00, 81.12it/s]
time = 0.09404134750366211
100%|███████████| 4/4 [00:00<00:00, 80.94it/s]
time = 0.09416794776916504
100%|███████████| 4/4 [00:00<00:00, 80.91it/s]
time = 0.0948340892791748

https://www.reddit.com/r/StableDiffusion/comments/17ecdab/we_are_now_at_10_frames_a_second_512x512_with/

[AIhasArrived]

Hello, is this related to sad talker, were you able to integrate it and make sad talker faster please? Please answer @aifartist

[AIhasArrived]

Dear @aifartist , you might be the person I need.
I am struggling for 3-4 weeks trying to find a way to get FAST lip sync, (and sad talker thing (where this post was written)),
My problem, processing is slow, you get faster results with LOWER resolution videos and lower resolution image input,
Other problem, can't seem to find a way to use all the GPU in all steps (sometimes you can use --batch_size) in some projects but I dont think it affects al the steps.
I tired many things (sad talker, wav2lip) and still searching.
Another problem is upscaling is very slow, and all these techs procude low resolutions outputs.

Can you reming me what is realtime open pose? (I remember openpose to be maybe something relatedx to .. control net with SD? no?)
In any case, I think I might be interested in your "science"/ discoveries, and wonder if can use this stuff can help me in my need?
(I also want to learn more AI/python stuff)

[AIhasArrived]

Hey @aifartist was my message too much for you?

[aifartist]

Why, because I didn't instantly reply to you? I've been 100% focused only learning the openpose->image thing to do a realtime video. Took me a full day to be able to rotate her right forearm in realtime. But now that I can do this I hope to soon have her twerking :-) Below is my first attempt from last night.

was my message too much for you?
Hmmm,...

Screencast.from.12-08-2023.07.48.37.PM.webm

[AIhasArrived]

lol I see you are having fun, I am pretty sure any knowledge can be used for anything, from fun to real practical use for all humans one day, and yes TBH i thought you might have been busy, so what did you think of my message?

[aifartist]

I couldn't understand everything in your longer post.
But, yes, this is related to control net.
FYI,
Control net is slow
T2I is a variation on CN which more lightweight and fast.
My question is whether it is fast enough for Realtime and it now seems like it might be.
When researching CN the articles seem to 99% talk about pose estimation which is not what I need.
Pose estimation is about taking an existing image and creating the "pose" or stick figure from the image. That can THEN be used to generate another image of a different person in that pose.
It is this second half I'm working on.