Inference and embeddings directly on web browser with wasm

[ngxson]

Hi everyone,

I've just made a WebAssembly binding for llama.cpp that can run completely on browser. It already have support for high-level API like completions, embeddings, as well as some low-level functions like sampling, (de)tokenization,...

Wrapper code is written in typescript, can be embedded inside a ReactJS project (there's an example on my repo).

Would be nice if someone can play around and make a complete RAG-in-browser demo :-)

Link to the project: https://github.com/ngxson/wllama

[phymbert]

Wow, nice job. I will give it a try. I see in the TODO: Guide: How to split gguf file?
We just discussed it in:

• GGUF huge files size / Split possible ? #6000

Are you working on something similar ?

[ngxson]

I did read your proposal this morning, thanks for bringing it up. Yeah it would be nice to split gguf and still loadable by llama.cpp without doing cat

The "split" feature that I mentioned in my repo is simply split & cat. I added it mostly for networking performance, since splitting file is a common technique when talking about web.

[phymbert]

FYI: https://huggingface.co/docs/hub/gguf

[ngxson]

Yeah thanks for the info. In fact, I work on gguf format when I was making the merge tool #5741 so I know more or less the file layout.

The tricky part is that tensor info holds the offset of tensor inside a file, so if we split the file into multiple parts, we cannot reuse the same metadata table (since offset is changed).

In #6000 I proposed to have separated gguf context (that means each gguf has its own table of tensor info), so the offset will be correct. At high level it should be a simple change, but I've got time to dive deep into that. If you're not actively working on this, please let me know so maybe I'll have a look this weekend or next week.

[phymbert]

II will ping you on the PR for support