Decompress .gz file in Azure Data lake

Question

How to decompress and read a .gz file which is in Azure data lake using c# asp.net

I have tried the following code but it results in an exception that.

Exception: Could not find a part of the path 'D:\xxxxxx\filename'.

public static void Main(string[] args)
    {
        // Obtain AAD token
        var creds = new ClientCredential(applicationId, clientSecret);
        var clientCreds = ApplicationTokenProvider.LoginSilentAsync(tenantId, creds).GetAwaiter().GetResult();

        // Create ADLS client object
        AdlsClient client = AdlsClient.CreateClient(adlsAccountFQDN, clientCreds);

        try
        {
            // Enumerate directory
            foreach (var entry in client.EnumerateDirectory("/Test/"))
            {
                try
                {
                    string filename =entry.Name;
                    using (Stream fileStream = File.OpenRead(filename), zippedStream = new GZipStream(fileStream, CompressionMode.Decompress))
                    {
                        using (StreamReader reader = new StreamReader(zippedStream))
                        {

                            // work with reader
                            reader.ReadLine();

                        }
                    }
                }
                catch (Exception ex)
                {

                }
            }
        }
        catch (AdlsException e)
        {
            PrintAdlsException(e);
        }

        Console.WriteLine("Done. Press ENTER to continue ...");
        Console.ReadLine();
    }

Answer 1

I got the solution. Instead of File.OpenRead(filename) we should use client.GetReadStream(entry.FullName).

The code is :

 foreach (var entry in client.EnumerateDirectory("/Test/"))
                {
                    StringBuilder lines = new StringBuilder();
                    try
                    {
                        using (Stream fileStream = client.GetReadStream(entry.FullName), zippedStream = new GZipStream(fileStream, CompressionMode.Decompress))
                        {
                            using (StreamReader reader = new StreamReader(zippedStream))
                            {
                                string line;
                                while ((line = reader.ReadLine()) != null)
                                {
                                    lines.AppendLine(line);
                                    Console.WriteLine(lines);
                                }
                            }
                        }
                    }

Answer 2

The built-in extractors (Text, Csv, Tsv) now natively support gzipped files so you do not have to do anything special other than read them:

@data =
    EXTRACT Timestamp DateTime,
            Event string,
            Value int
    FROM "/input/input.csv.gz"
    USING Extractors.Csv();

This also works for custom extractors:

@data =
    EXTRACT Timestamp DateTime,
            Event string,
            Value int
    FROM "/input/input.csv.gz"
    USING  new USQLworking.MyExtractor();

See here for further note from Michael Rys.