New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
More explicit prompting to help smaller models #562
base: main
Are you sure you want to change the base?
Conversation
…ool argument. Repeat the same instruction each time.
I'll need to run the benchmarks on this one, so might take a little longer to merge |
I tested this with mixtral-8x7b and the Action Input was now correct consistently where before it was more like 60% success rate. llama3-8b still failed to close the dictionary.
|
I spoke too soon, with further testing I am still seeing tool use failures with
|
Thanks David. The prompt style from phidata works quite well on small models. This would move away from crewai's more conversational style, so i didn't suggest it. ''' The model is being clever and adding meaningful sublists or dicts in valid JSON style - not what was asked for but generally useful and creatively coercable. |
Some further tests when using phidata style prompts. When using small models, we may be condemned to barbary. {'dolphin-llama3:8b': {'elapsed': 54.291745448112486, |
great news, in this new version we will add the opportunity for people to overwrite all the inner prompts, not saying we shouldn't benchmark this still, but something that will help with individual models |
Hi there. Prompts are at the core of crewAI's ability to orchestrate models to use tools correctly.
When testing with smaller models (In my case, variants of mistral, llama3 and phi3), often tool parameters were missing curly braces, or included into the tool name, resulting in multiple error sequences.
These minor changes to the phrasing should increase reliability.
They have NOT been tested on large models, though there is a good chance that large models are able to understand without being so explicit.