-
Notifications
You must be signed in to change notification settings - Fork 101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error: http: request body too large (recovered by wazero) #673
Comments
I just opened a PR to update the CLI to add the Other than that the only other limit I can think of is the 32-bit address space. Also, this setting only applies to calls to |
Now I get a random Steps :
from what I gathered I ran a quick check and I might be missing things, my guess the excessive memory usage happens in extism/runtime/src/current_plugin.rs Line 103 in fc22412
// +1gb allocated in `http_request`, then in
pub fn memory_new<'a, T: ToBytes<'a>>(&mut self, t: T) -> Result<MemoryHandle, Error> {
let data = t.to_bytes()?; //<= +700mb ?
let data = data.as_ref();
if data.is_empty() {
return Ok(MemoryHandle::null());
}
let handle = self.memory_alloc(data.len() as u64)?; //<= +700mb ?
let bytes = self.memory_bytes_mut(handle)?;
bytes.copy_from_slice(data.as_ref()); //<= +700mb ?
Ok(handle)
}
//=> ~4gb and I am probably missing out some other allocations. I didn't have time to run Valgrind to confirm, in any case, this is not urgent but I think it's worth following up as there is room for improvement that would benefit the memory management and speed of extism runtime. |
Thanks for testing it out! Your example does download the full ISO on my machine, but memory use seemed to peak at ~4GB, so there is definitely some room to improve memory use around large HTTP responses. I will investigate a little further today. |
@mtb0x1 we merged a solution for this last week, just want to follow up to make sure you saw & see if it works for you! |
I tested with the change in http plugin, memory usage did improve and peak usage did go down from +4Gb to 2.7Gb. |
Command to reproduce :
Result :
Expected Result :
iso file downloaded entirely, if not a warning/error message explaining the reason.
in
runtime/src/pdk.rs
, there is a limit set to 50MB (?).extism/runtime/src/pdk.rs
Lines 240 to 242 in 1a083f6
My question I guess is, beyond the TODO to configure the limit, is there a hard-limit that we should worry about either due to Extism or WASI design.
The text was updated successfully, but these errors were encountered: