Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storing values larger than 2 GB #574

Open
mitar opened this issue Jan 1, 2024 · 0 comments
Open

Storing values larger than 2 GB #574

mitar opened this issue Jan 1, 2024 · 0 comments

Comments

@mitar
Copy link
Contributor

mitar commented Jan 1, 2024

Is your feature request related to a problem? Please describe.

I would want to store values larger than 2 GB.

Describe the solution you'd like

Be able to configure SegmentSize to a value larger than 2 GB and then be able to store larger files. Currently this is not possible because there is a hard-coded limit in value size to MaxInt32 value. (In fact, I think there is a bug and it should really be MaxUint32 on 64bit systems.)

I suggest the limit is raised to MaxInt64 on 64bit systems.

Describe alternatives you've considered

I could split large value into multiple smaller values on my side and then store all of those splits manually and recombine them.

Additional context

It seems to me that this is just to support mmap on 32 bit systems and there is really no design limitation why not to support larger values? On my system, I have:

$ grep -m1 "address sizes" /proc/cpuinfo 
address sizes	: 39 bits physical, 48 bits virtual

So a bit less than 2^47 (128 TiB) could be the max limit for mmap on my system.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant