Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check failed: (!free_page_ids_.empty()) is false: The KV cache is full. No page can be allocated. #385

Closed
pilosof opened this issue Apr 29, 2024 · 1 comment
Closed

Check failed: (!free_page_ids_.empty()) is false: The KV cache is full. No page can be allocated. #385

pilosof opened this issue Apr 29, 2024 · 1 comment

Comments

@pilosof
Copy link

pilosof commented Apr 29, 2024 •
edited

when prompting to summarize long article or long chat

  • system: macbook pro m1
  • model: llama3
@CharlieFRuan CharlieFRuan mentioned this issue Jun 4, 2024
Merged
CharlieFRuan added a commit that referenced this issue Jun 5, 2024
@CharlieFRuan
[OpenAI] Handle exceeding context window size, rename max_gen_len to …
6a7fe3e 
…max_tokens (#457)

- **[Breaking]** Rename `max_gen_len` to `max_tokens` in
`ChatCompletionRequest`
- Remove reading `max_gen_len` from `mlc-chat-config.json`, hence remove
it from `ChatOptions`
- Remove reading `mean_gen_len` and `shift_fill_factor` from
`mlc-chat-config.json`, hence:
  - Remove these two fields from `ChatOption`
- Throw error when the input prompt tokens exceed `context_window_size`,
prompting user to truncate input, or increase `context_window_size`, or
use sliding window
- Finish reason with `length` when decoding exceed `context_window_size`
(when `prompt_size + max_tokens > context_window_size`)
- Hence this error is never shown to user:
#385

This change is needed in order to be compatible with new
`mlc-chat-config.json` in hugginface due to
mlc-ai/mlc-llm#2493
@CharlieFRuan
Copy link
Contributor

Thanks for reporting the issue. This error should not be displayed anymore after #457, integrated into 0.2.42 and later.

If the request's length exceeds KVCache's context window size, we will throw error. If during decode, the number of generated tokens exceed the context window size, we end the response with finish reason "length".

To fix, try one of the three options:

@tqchen tqchen closed this as completed Jun 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pilosof @tqchen @CharlieFRuan