You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
not surport in-flight batching, which will be a great concern since this feature greatly improve thorouput
temperature == 0, how about temperature > 0?
kv_cache, I guess kv_cache needed recompute because generate_tokens will > 1
My biggest concern is whether Medusa2 conflicts with in-flight batching?
The text was updated successfully, but these errors were encountered:
Ageliss
changed the title
[Dynamic Batching] Concers about what features are not supported using Medusa
[Dynamic Batching] Concerns about whether features are not supported using Medusa
Feb 21, 2024
I checked the TRT-LLM but found something confusing. There are some features not supported:
My biggest concern is whether Medusa2 conflicts with in-flight batching?
The text was updated successfully, but these errors were encountered: