How to control the size of work buffer?

[wilderfield]

When we run a matrix multiplication op, we are given compute params, some of which are a work buffer and work buffer size.

Where can I control that?

I actually want the size to be proportional to the size of one of the input tensors, however that may not be feasible, because the inputs are known to be dynamic in size.

(I am trying to avoid allocating extra memory at runtime by using the work buffer).

[ggerganov]

Take a look at the simple example and see if it makes things more clear. Let us know if you have more questions

[wilderfield]

What I don't get, is in the simple example, APIs are used to build a model, and the context size is manually established.

If I am running a real GGUF model, I don't know what kind of space was reserved for the work buffer seen by an individual ggml op.

I also don't quite understand what is the purpose of a work buffer?