CODEWITHSUNDEEP
c#azure-functionsparqueteventtriggerchoetl
Amey Pimpley
Amey Pimpley

Reputation: 31

Using Event trigger Azure function, how to convert csv file to parquet file using c#

I want to get a file in azure blob storage and using event trigger function, I want to convert this csv file into parquet format and upload it to another blob storage. I tried to use choparquet reader, but it takes a lot of time to convert the file to parquet format and when a large files comes, few events are missed from the queue and not all files are this converted in to parquet format. I am using c#

Can anyone help in simplified approach to this problem? The code I am working is

using (Stream blobStream = await blockBlob.OpenWriteAsync(accessCondition, null, null))
{
    using (var reader = new StreamReader(longestFile.Open(), Encoding.UTF8))
    using (var r = ChoCSVReader.LoadText(reader.ReadToEnd()).WithMaxScanRows(2).WithDelimiter("\t"))
    {
        log.LogInformation($"Inside csv reader");
        using (var w = new ChoParquetWriter(blobStream))
        {
            log.LogInformation($"Inside parquet writer");
            w.Write(r);
            w.Serialize();
            w.Close();
        }
    }
}

Upvotes: 0

Views: 610

Answers (0)

Related Questions