Add the option of saving in parquet instead of arrow

[arita37]

Feature request

In dataset.save_to_disk('/path/to/save/dataset'),

add the option to save in parquet format

dataset.save_to_disk('/path/to/save/dataset', format="parquet"),

because arrow is not used for Production Big data.... (only parquet)

Motivation

because arrow is not used for Production Big data.... (only parquet)

Your contribution

I can do the testing !

[Dref360]

I think Dataset.to_parquet is what you're looking for.

Let me know if I'm wrong

[arita37]

No, it does not save the metadata json. We have to recode all meta json load/save with another custome functions. save_to_disk and load should have option with “Parquet” instead of “arrow” since “arrow” is never user for production (only parquet). Thanks !

…

On May 17, 2024, at 5:38, Frédéric Branchaud-Charron ***@***.***> wrote:  I think Dataset.to_parquet is what you're looking for. Let me know if I'm wrong — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.