New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: openai api compatible model presets (profiles) #665
Comments
Ollama.ai is not directly associated with this project. Your feature request belongs with them. |
Thanks for taking a look. I already implemented this and it works for me. See the referenced PR. |
OK first, thanks for adding features. It's awesome to have a community that's adding to a project. Per the PR, I have some suggestions that I'll post over there. |
Changed my mind, I'll post here: Modelfiles are an Ollama.ai-specific feature to describe settings that belong to a specific model. OpenAI doesn't have such a thing because most of the provided functionality it is wrapped up in the product offering. Of course we live in a world where the OpenAI API is becoming the standard even for interacting with custom models and local LLM servers. That leaves a functionality hole that you're trying to address here. However, I think the over-arching feature request (added metadata for non-local, non-ollama.ai models) needs to be thought of a bit more holistically. For example, in the UI, we set an Ollama server and optionally and OpenAI key and endpoint. the WebUI then gives you access to both. I think the right way to solve this issue is to unify the idea of external LLM servers much like how LiteLLM handles it. (That and even Ollama.ai is working on adopting OpenAI's API.) Your PR provides metadata to all in an agnostic way but maintains the type-difference segregation. The whole idea of multiple models/servers/configurations is tied up in other feature requests that should be considered since they all would (should?) share the same solution: #278 #619 |
I also feel like the Modelfile format is the wrong format for its purpose. It's patterned after Dockerfile (possibly since the ollama.ai author previously worked on docker). But Dockerfile is the way it is because each line represents a progression. This is not a thing in Modelfile. I feel like we should have our own custom file to manage this metadata. TOML or YAML (but I haven't put too much thought into it). |
@explorigin, you've precisely captured the essence of the thoughts I've been struggling to express. Modelfile is designed with Ollama's specific needs and objectives in mind. Merely suggesting to incorporate OpenAI into it without a thorough strategy might not be the most effective approach to address the underlying issue. |
Thanks guys for thinking about this. To get a character from ollamahub, but use it with an openAI API (in my case together.ai) So maybe the request is this: Make ollamahub models also usable with openAI API compatible services, not just ollama. What do you think? |
Ollamahub is supposed to provide access to anything that the user can download via Ollama.ai...so allowing it to reference any custom model will likely lead to a bad user-experience if that model in question is not available. We certainly don't want modelfiles pointing to together.ai endpoints. LMStudio references models by huggingface path (e.g. TheBloke/CapybaraHermes-2.5-Mistral-7B-GGUF). This feels like the right way to go w.r.t. applying specific settings to specific models. It's quantization-agnostic while remaining model-specific. Ollama.ai tries a little too hard to be docker with it's So what we want is a sort of configuration file that:
Anything else? Definitely think we should consider what's in LMStudio's Model configuration dialog. |
Absolutely, I concur with everything above. However, I believe it's crucial that we extend our exploration to the broader community to ensure we're not overlooking any existing generalized solutions to this problem that we could potentially endorse. Let's avoid bike-shedding another standard if possible... (Obligatory xkcd reference: https://xkcd.com/927/) |
It's not a standard if it's not applied everywhere. 😁 # 15 here we come! |
TODO
|
I feel like trying to find/create a new general model format for everything is boiling the ocean here. Essentially there are two problems, that I can see:
I think that:
|
Is your feature request related to a problem? Please describe.
As the modelfiles from the hub are pretty good, it would be nice if we could use also them with external models from openAI api compatible services.
Describe the solution you'd like
Option A:
When adding a modelfile, we could check that it should use a model from the external provider and select it.
Option B:
When selecting a modelfile, we should be able to exchange then the actual model to one from the external service.
Describe alternatives you've considered
Setting the default prompt according to the modelfile works for now, just the profile pick and name is wrong and the extra settings.
Additional context
My ollama instance has limited vram available. External services can provide bigger models. I think it would be good if the same characters would be usable when using an external service.
The text was updated successfully, but these errors were encountered: