Return buffer while processing Stream

Question

So I have a file upload form which (after uploading) encrypts the file and uploads it to an S3 bucket. However, I'm doing an extra step which I want to avoid. First, I'll show you some code what I am doing now:

using (MemoryStream memoryStream = new MemoryStream())
{
    Security.EncryptFile(FileUpload.UploadedFile.OpenReadStream(), someByteArray, memoryStream);
    memoryStream.Position = 0; // reset it's position
    await S3Helper.Upload(objectName, memoryStream);
}

My Security.EncryptFile method:

public static void EncryptFile(Stream inputStream, byte[] key, Stream outputStream)
{
    CryptoStream cryptoStream;

    using (SymmetricAlgorithm cipher = Aes.Create())
    using (inputStream)
    {
        cipher.Key = key;
        // aes.IV will be automatically populated with a secure random value
        byte[] iv = cipher.IV;

        // Write a marker header so we can identify how to read this file in the future
        outputStream.WriteByte(69);
        outputStream.WriteByte(74);
        outputStream.WriteByte(66);
        outputStream.WriteByte(65);
        outputStream.WriteByte(69);
        outputStream.WriteByte(83);

        outputStream.Write(iv, 0, iv.Length);

        using (cryptoStream =
            new CryptoStream(inputStream, cipher.CreateEncryptor(), CryptoStreamMode.Read))
        {
                cryptoStream.CopyTo(outputStream);
        }
    }
}

The S3Helper.Upload method:

public async static Task Upload(string objectName, Stream inputStream)
{
    try
    {
        // Upload a file to bucket.
        using (inputStream)
        {
            await minio.PutObjectAsync(S3BucketName, objectName, inputStream, inputStream.Length);
        }
        Console.Out.WriteLine("[Bucket] Successfully uploaded " + objectName);
    }
    catch (MinioException e)
    {
        Console.WriteLine("[Bucket] Upload exception: {0}", e.Message);
    }
}

So, what happens above is I'm creating a MemoryStream, running the EncryptFile() method (which outputs it back to the stream), I reset the stream position and finally reuse it again to upload it to the S3 bucket (Upload()).

The question

What I'd like to do is the following (if possible): directly upload the uploaded file to the S3 bucket, without storing the full file in memory first (kinda like the code below, even though it's not working):

await S3Helper.Upload(objectName, Security.EncryptFile(FileUpload.UploadedFile.OpenReadStream(), someByteArray));

So I assume it has to return a buffer to the Upload method, which will upload it, and waits for the EncryptFile() method to return a buffer again until the file has been fully read. Any pointers to the right direction will be greatly appreciated.

Answer 1

What you could do is make your own EncryptionStream that overloads the Stream class. When you read from this stream, it will take a block from the inputstream, encrypt it and then output the encrypted data.

As an example, something like this:

public class EncrypStream : Stream {

    private Stream _cryptoStream;
    private SymmetricAlgorithm _cipher;

    private Stream InputStream { get; }
    private byte[] Key { get; }

    public EncrypStream(Stream inputStream, byte[] key) {
        this.InputStream = inputStream;
        this.Key = key;
    }

    public override int Read(byte[] buffer, int offset, int count) {

        if (this._cipher == null) {
            _cipher = Aes.Create();
            _cipher.Key = Key;

            // aes.IV will be automatically populated with a secure random value
            byte[] iv = _cipher.IV;

            // Write a marker header so we can identify how to read this file in the future
            // @TODO Make sure the BUFFER is big enough...
            var idx = offset;
            buffer[idx++] = 69;
            buffer[idx++] = 74;
            buffer[idx++] = 66;
            buffer[idx++] = 65;
            buffer[idx++] = 69;
            buffer[idx++] = 83;

            Array.Copy(iv, 0, buffer, idx, iv.Length);
            offset = idx + iv.Length;

            // Startup stream
            this._cryptoStream = new CryptoStream(InputStream, _cipher.CreateEncryptor(), CryptoStreamMode.Read);
        }

        // Write block
        return this._cryptoStream.Read(buffer, offset, count);
    }

    protected override void Dispose(bool disposing) {
        base.Dispose(disposing);

        // Make SURE you properly dispose the underlying streams!
        this.InputStream?.Dispose();
        this._cipher?.Dispose();
        this._cryptoStream?.Dispose();
    }

    // Omitted other methods from stream for readability...
}

Which allows you to call the stream as:

using (var stream = new EncrypStream(FileUpload.UploadedFile.OpenReadStream(), someByteArray)) {
     await S3Helper.Upload(objectName, stream);
}

As I notice your upload method requires the total bytelength of the encrypted data, you can look into this post here to get an idea how you would be able to calculate this.

(I'm guessing that the CryptoStream does not return the expected length of the encrypted data, but please correct me if I'm wrong on this)