Huge max_heap_size compared to the actual data size

[IvanIvanoff]

Environment

• Elixir version (elixir -v): Elixir 1.15.0-dev (e0c1084) (compiled with Erlang/OTP 25)
• Absinthe version: 1.7.1 . I've tested also with 1.6.8 and 1.5.5

Actual behavior

A few days ago our application started getting killed with Out of memory error.
What we found is that for some queries, even if the size of the response is small, the memory consumed by the process that processes the GraphQL query is huge. When multiple such queries happen concurrently, the RAM usage jumps from 1Gb to over 6Gb. As a temporary solution we set the :max_heap_size flag in the resolver before processing the query, so these queries can be killed instead of killing the whole application.

Here is a project that reproduces the issue: https://github.com/IvanIvanoff/absinthe_memory

In this project, we observe that when the HTTP response is small in size at 828 kB, Absinthe peaks at over 50 Mb of RAM usage when processing it.

Expected behavior

The peak memory usage should not be so big.

[benwilson512]

Happy to look into it. I suspect that using persistent term will help. Overall though for bulk data like timeseries data the fact the remains that there is going to be a fair bit over overhead, both in processing and memory, compared to raw JSON. Absinthe is going to type annotate and type check every single value and if you want to move thousands of values, that adds up.

[benwilson512]

Overall I would suggest that you treat bulk data transfers a bit like you'd treat images or binary data in GraphQL: Link to them. We have a bulk data controller that takes a set of signed parameters and returns JSON. Then in our GraphQL we return links to those endpoints with the signed params embedded.