LFX Workspace: Integrate whisper.cpp as a new WASI-NN backend #3170

[rum1887]

Summary

WasmEdge supports PyTorch, TensorFlow Lite, llama.cpp, and more NN backend. Dealing with the voice-to-text is a big thing that we want to achieve. To make it possible, we would like to integrate whisper.cpp, a port of OpenAI's Whisper model in C/C++ as a new WASI-NN backend.

Details

Milestones

Progress: here

Week 1-2

• Review the llama.cpp project

Week 2-4

• Complete the initial structure of the plugin
• Unit tests

Week 5-6

• Testing the plugin
• Develop examples

Midterm PR:

Week 7-8

• Develop more examples and enhance the plugin

Week 9-10

• Tutorials, Blogging and publicising

Week 11-12

• Enhancements

Final Work Report:

Appendix

No response

[hydai]

Review the llama.cpp project

llama.cpp and whisper.cpp is different. Please make sure you are working on the correct project.

[rum1887]

Review the llama.cpp project

llama.cpp and whisper.cpp is different. Please make sure you are working on the correct project.

yes, I included llama.cpp because whisper.cpp also uses the ggml backend.

[rum1887]

CODE UPDATE [1]:
I have written a basic file( whisper.cpp & whisper.h ) to load the model using default whisper parameters.

[hydai]

Please update the progress on this issue. Nothing has been updated in a long time. And the last update of your PR was two weeks ago.

[rum1887]

Update:
Working on a small example + unit case for loading with parse. Will update the pr by the end of this week and discuss over office hours

[hydai]

Hi @rum1887
The issue you mentioned in the office hours is fixed now: #3379
Please update to the latest master branch to apply the patch.

[hydai]

Hi @rum1887
I am still not receiving your progress and any new commits inside your PR. Please update them in the recent days.

[hydai]

This is a final notification. We still haven't received your PR or updates for the final evaluation.

[hydai]

We tried to contact the mentee. However, we didn't receive any further updates from Discord, GitHub issue, and PR. I am closing this because the mentee is no longer working on it.