Models for testing inference / sampling

[danemadsen]

I see there's a number of models for testing tokenization here However i would like a model for testing out my implementation of llama.cpp into my front end. Is anyone aware of such a model?

basically any model that would work with simple.cpp and is relatively small (less than 100mb so GIT LFS isn't needed)

[ggerganov]

You can use tinyllamas: https://github.com/ggerganov/llama.cpp/tree/master/examples/convert-llama2c-to-ggml

[danemadsen]

Ahh ok. They are indeed small but still too large for a test model unfortunately. I don't really care about the output, it could only output EOS tokens for example. Just that it would be good to have a model I can use in my CI to test things are working.

[ggerganov]

The stories260k is 1MB - you can hardly get smaller than that. We use it in the CI as well:

llama.cpp/.github/workflows/build.yml

Lines 236 to 246 in 24affa7

	- name: Test llama2c conversion
	id: llama2c_test
	run: |
	cd build
	echo "Fetch tokenizer"
	wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories260K/tok512.bin
	echo "Fetch llama2c model"
	wget https://huggingface.co/karpathy/tinyllamas/resolve/main/stories260K/stories260K.bin
	./bin/convert-llama2c-to-ggml --copy-vocab-from-model ./tok512.bin --llama2c-model stories260K.bin --llama2c-output-model stories260K.gguf
	./bin/main -m stories260K.gguf -p "One day, Lily met a Shoggoth" -n 500 -c 256

[danemadsen]

Ahh OK that might be more what I'm looking for. Thanks for the advice