Custom Model Generator

[shanyssys]

Lately I've run into models that are having issues loading into Ollama, they usually end up with "out of memory". For example 7b models like Gemma using 3070ti.

The solution to load 7b models into ollama is to restrict the number of layers which pytorch is loading into the gpu,
this is done by using a flag PARAMETER num_gpu 25 in the Modelfile.

which will look like this when ollama runs the model in log:
llm_load_tensors: offloading 25 repeating layers to GPU

As a feature request, I'd like to see a "Set layers for model X" in Open-WebUI which then creates a new model file, uses the ollama create flag and have it all listed. this will help benchmark older cards with less gpu than just rely on the cpu with num_gpu 0 flag.

[justinh-rahb]

To clarify, you mean adding the field here in the Modelfile generator?

[Gally-Youko]

I want this too
#1749

[shanytc]

Yeah, if possible to add the num_gpu.
If it's 0 it means cpu, otherwise it will load it the specified amount layers to the gpu

[gigascake]

Using this method, Enable Load Model to CPU(DRAM) + VRAM?