Replies: 3 comments
-
Now you find the original link and this ticket :) A duckduckgo query gives me pictures of parquetry. A short intro or link would be helpful. |
Beta Was this translation helpful? Give feedback.
-
Perhaps this medium article helps? |
Beta Was this translation helpful? Give feedback.
-
Thanks! - that's useful. I guess it's an Oracle concept. I'm still a little fuzzy on exactly what I need to do in practice building parquet files to get the most out of duckdb. For example, focusing on construction of the index, or making the row groups align with the primary keys intuitively feels like it would help, but I don't see enough in docs to really know what to do. It's impressively fast in practice so maybe I'm overthinking things. |
Beta Was this translation helpful? Give feedback.
-
After reading the duckdb documentation I found the concept of "zonemaps" there:
https://duckdb.org/docs/data/parquet/overview.html#partial-reading
This seems very compelling to me, but I've been unable to google "parquet zonemap" and find anything other relevant than the original link.
Is there some equivalent? Partitioning a parquetfile like you do with pyarrow? Where can I go to understand more about the concept and how to store files to actually contain these "zonemaps"?
Beta Was this translation helpful? Give feedback.
All reactions