Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for max_submission_tokens to presets #322

Open
joshcho opened this issue Jul 11, 2023 · 3 comments
Open

Add support for max_submission_tokens to presets #322

joshcho opened this issue Jul 11, 2023 · 3 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@joshcho
Copy link

joshcho commented Jul 11, 2023

Describe the bug
In the config, unclear as to how to set max_submission_tokens (in the preset or in the config?). Also, preset-save and -load do not operate correctly with max_submission_tokens. Possible that max_submission_tokens is not associated with model preset, but then this is not documented.

To Reproduce
1)
Set config file for the preset:

metadata:
  name: gpt-4
  provider: chat_openai
model_customizations:
  max_submission_tokens: 2000
  model_name: gpt-4
  n: 1
  request_timeout: null
  streaming: true
  temperature: 1.0

Says Invalid key.

  1. Set max-submission-tokens via
    /max-submission-tokens 2000
  2. Save preset via /preset-save my-preset
  3. Load preset via /preset-load my-preset

Expected behavior
It should work as expected, loading the customization.

@thehunmonkgroup
Copy link
Collaborator

max-submission-tokens is not currently supported in preset configuration, it is a runtime only setting.

I would look at a PR for this feature if it was well considered.

@thehunmonkgroup thehunmonkgroup added enhancement New feature or request help wanted Extra attention is needed labels Jul 11, 2023
@thehunmonkgroup thehunmonkgroup changed the title Setting max_submission_tokens Add support for max_submission_tokens to presets Jul 11, 2023
@joshcho
Copy link
Author

joshcho commented Jul 12, 2023

I think even in runtime, max-submission-tokens doesn't quite work. I am getting KeyError('content') when my token count goes above max-submission-tokens.

@thehunmonkgroup
Copy link
Collaborator

I think even in runtime, max-submission-tokens doesn't quite work

Yep, good catch. Fixed in the 0.14.1 release: https://github.com/llm-workflow-engine/llm-workflow-engine/releases/tag/v0.14.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants