New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Values in json_schema_extra get sorted automatically during model_json_schema
#508
Comments
i've always used chain_of_thought and never noticed, will talk to the pydantic team |
I also was putting `user` as a key to simulate the user request and then
the response. The QA example didn’t seem practical because then it requires
the user prompt in the model? Am I missing something there?
…On Sat, Mar 16, 2024 at 5:05 PM Jason Liu ***@***.***> wrote:
i've always used chain_of_thought and never noticed, will talk to the
pydantic team
—
Reply to this email directly, view it on GitHub
<#508 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFMAUL54ZUIE4GC2NULGXLYYTM3ZAVCNFSM6AAAAABEZSVOJCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBSGE4TKNBQHE>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
can you give a code snippet? i can't see what youlre thinking about |
Something like this:
I ended up going with something like this:
which seems to work well. this is the sorting issue I was talking about:
this breaks CoT because the model thinks it' should output the In debugging all of this, I also noticed that the schema that gets sent includes references for everything. For simple schemas where you're not even re-using the models this increases the token count + makes the usage more unreliable from what I can tell. ie:
turns into:
the LLM has to parse the references correctly along with do the task which I was getting poor results with. I worked around the sorting + this referencing issue with this:
which results in:
depends on the schema and how often you're referencing the same class, but I got significantly better results with this. |
What Model are you using?
Describe the bug
This isn't a bug with instructor per se but any examples you add to json_schema_extra get sorted: pydantic/pydantic#7580. This can impact the quality of the generation if you have "thought" for instance to trigger CoT, it will get output at the end and then steer the model towards doing that too.
There should be an option to avoid this in pydantic but thought it was worth noting in the example since this threw me off.
To Reproduce
Steps to reproduce the behavior, including code snippets of the model and the input data and openai response.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
The text was updated successfully, but these errors were encountered: