Skip to content

Latest commit

 

History

History
71 lines (61 loc) · 2.33 KB

README.md

File metadata and controls

71 lines (61 loc) · 2.33 KB
Raw

LipSync

Lips Synchronization (Wav2Lip).

Install

git clone git@github.com:mowshon/lipsync.git
cd lipsync
python setup.py install

Download the weights

Model Description Link to the model
Wav2Lip Highly accurate lip-sync Link
Wav2Lip + GAN Slightly inferior lip-sync, but better visual quality Link

Project structure

└── project-folder
   ├── cache/
   ├── main.py
   ├── wav2lip.pth
   ├── face.mp4
   └── audio.wav

Example

from lipsync import LipSync


lip = LipSync(
    checkpoint_path='wav2lip.pth',  # Downloaded weights
    nosmooth=True,
    cache_dir='cache'  # Cache directory
)

lip.sync(
    'face.mp4',
    'audio.wav',
    'output-file.mp4'
)

License and Citation

The software can only be used for personal/research/non-commercial purposes. Please cite the following paper if you have use this code:

@inproceedings{10.1145/3394171.3413532,
    author = {Prajwal, K R and Mukhopadhyay, Rudrabha and Namboodiri, Vinay P. and Jawahar, C.V.},
    title = {A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild},
    year = {2020},
    isbn = {9781450379885},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3394171.3413532},
    doi = {10.1145/3394171.3413532},
    booktitle = {Proceedings of the 28th ACM International Conference on Multimedia},
    pages = {484–492},
    numpages = {9},
    keywords = {lip sync, talking face generation, video generation},
    location = {Seattle, WA, USA},
    series = {MM '20}
}

Acknowledgements

Parts of the code structure is inspired by this TTS repository. We thank the author for this wonderful code. The code for Face Detection has been taken from the face_alignment repository. We thank the authors for releasing their code and models.