Allow Instructor to work on other LLM's

[phatlast96]

Is your feature request related to a problem? Please describe.
I love Instructor. Is it possible to have it work on other LLM's with varied API. In the end, all the API does is just aggregate it into one prompt right. Theoretically, it should work.

Describe the solution you'd like
Have a way for Instructor to output a large prompt to any LLM's. Since LLM's take prompts in the end, all the work that Instructor does to organize information should be available as a large prompt.

Similar to a compiler, it'd be great if Instructor "compiles" the Pydantic and all the validations to make it smooth for any LLM. Instructor should take a function, that contains the prompt and outputs generated answer, to be able to port Instructor's features.

Describe alternatives you've considered

• interface with other LLM via function wrapper

Additional context

• I'm trying to host an LLM on my machine and it'd be great to have combine that with Instructor. I can wrap the REST API around a Python function, but I was wondering if I can use that function in Instructor.

[jxnl]

are you using litellm or ollama?

[phatlast96]

I'm using Anthropic Claude API and I'm trying to host Mistral-8x7-Instruct. Because of costs, I intend to have the Mistral-8x7-Instruct to do the validation on the Claude via hosting on EC2 machine. The reason is that I'm trying to prompt for every one of my table row (200 rows in total), which get expensive quite fast if using OpenAPI, especially with validation since the output of the query LLM would be an additional input for the validation LLM.

I think Instructor is a genius invention with the pydantic validation and LLM validation, which has numerous use cases. It makes it easy to reduce hallucination and stick to fact-based use cases. Instructor also makes it 10x easier to turning unstructured data into structured.

[phatlast96]

Looking at litellm, how would that work with Instructor?

There's completion for litellm. You cannot do client = instructor.patch(OpenAI(api_key=settings.OPENAI_API_KEY), mode=instructor.Mode.JSON) as in instructor nor can you do client.chat.completions.create.

[jxnl]

try wrap_completion from instructor.patch

[jxnl]

Super busy right now, but I'll try to do something... This weekend.

[broomva]

How about Together's AI implementation? It should've worked out of the box, right? https://docs.together.ai/docs/integrations

[AndromedaPerseus]

It did not work for me, and I tried it with a couple of providers that use the OpenAI endpoint wrapper to instantiate the LLM. They all return with an error that I posted yesterday.

Technically, it should work out of the box.

[jxnl]

@broomva @EnkiEnil does this? just copy pasting stuff since i dont have an api key.

import openai
import instructor

system_content = "You are a travel agent. Be descriptive and helpful."
user_content = "Tell me about San Francisco"

client = openai.OpenAI(
    api_key="TOGETHER_API_KEY",
    base_url="https://api.together.xyz/v1",
    )
    
 client = instructor.patch(client. mode=instructor.Mode.MD_JSON)
    
model = client.chat.completions.create(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    messages=[
        {"role": "system", "content": system_content},
        {"role": "user", "content": user_content},
    ],
    response_model=Model.
    temperature=0.7,
    max_tokens=1024,
)
assert isinstance(model, Model)