CODEWITHSUNDEEP
c#csv.net-coreparquet
Elliott Barinberg
Elliott Barinberg

Reputation: 185

How to convert parquet file to CSV using .NET Core?

I have a parquet file and I am trying to convert it to a CSV file, it seems as though most recommend using Spark, however I need to use C# to accomplish this task, specifically I need to use .NET Core 3.0.

Its tricky because parquet is columnar data which is making it annoying to convert to CSV...

I have tried loading it into a datatable but I dont like that solution because I need the entire file in memory and I am losing certain records somehow.

I am using parquet.net but I am open to any other parquet library that works on .net core/standard

Thank you in advance.

Upvotes: 1

Views: 8026

Answers (2)

Cinchoo
Cinchoo

Reputation: 6322

With Cinchoo ETL - an open source library, you can convert Parquet file to CSV easily.

Install Nuget package

install-package ChoETL.Parquet

Sample code

using ChoETL;

StringBuilder csv = new StringBuilder();
using (var r = new ChoParquetReader(@"*** Your Parquet file ***")
    .ParquetOptions(o => o.TreatByteArrayAsString = true)
    )
{
    using (var w = new ChoCSVWriter(csv)
        .WithFirstLineHeader()
        .UseNestedKeyFormat(false)
        )
        w.Write(r);
}

Console.WriteLine(csv.ToString());

For more information, please visit codeproject article.

Upvotes: 3

Alex KeySmith
Alex KeySmith

Reputation: 17091

I haven't given it a shot, but I wonder whether you could leverage / abuse the Microsoft Spark SQL libraries to your benefit.

There's

DataFrameReader.Parquet(String[])

https://learn.microsoft.com/en-us/dotnet/api/microsoft.spark.sql.dataframereader.parquet?view=spark-dotnet

And also:

DataFrameWriter.Csv(String) Method

https://learn.microsoft.com/en-us/dotnet/api/microsoft.spark.sql.dataframewriter.csv?view=spark-dotnet#Microsoft_Spark_Sql_DataFrameWriter_Csv_System_String_

I wonder whether you could use a DataFrame as an in memory intermediary.

It's just a guess at the moment as your question intrigued me, perhaps I'll give it a shot once I've got some sleep. :-)

Upvotes: 1

Related Questions