How to Mock aws s3 SelectObjectContentAsync with .net using moq

Question

I am trying to mock SelectObjectContentAsync. I got examples for Java and go, but not for c#. I am getting the below error. I am guessing my setup is incorrect. I was able to mock "GetObjectAsync" before.

System.Net.Http.HttpRequestException : InternalServerError - Internal Server Error: Amazon.S3.Model.S3EventStreamException: Error. ---> Amazon.Runtime.EventStreams.EventStreamChecksumFailureException: Message Prelude Checksum failure. Expected 1382380405 but was -1954053792 at Amazon.Runtime.EventStreams.Internal.EventStreamDecoder.ProcessPrelude(Byte[] data, Int32 offset, Int32 length) at Amazon.Runtime.EventStreams.Internal.EventStreamDecoder.ProcessData(Byte[] data, Int32 offset, Int32 length) at Amazon.Runtime.EventStreams.Internal.EventStream2.ReadFromStream(Byte[] buffer) at Amazon.Runtime.EventStreams.Internal.EnumerableEventStream2.GetEnumerator()+MoveNext() --- End of inner exception stack trace --- at Amazon.Runtime.EventStreams.Internal.EnumerableEventStream`2.GetEnumerator()+MoveNext()

Below is my code which sets up mock s3 client for unit testing. I am setting up mock for the method "SelectObjectContentAsync"

using Amazon.S3;
using Amazon.S3.Model;
using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Moq;
class MockIAmazonS3
{
    private static Mock<IAmazonS3> GetMockS3()
    {
        var mockS3Client = new Mock<IAmazonS3>();            
        mockS3Client.Setup(x =>
            x.SelectObjectContentAsync(It.IsAny<SelectObjectContentRequest>(), It.IsAny<CancellationToken>()))
        .Returns(new Func<SelectObjectContentRequest, CancellationToken, Task<SelectObjectContentResponse>>(GetSelectObjectContentResponse));

        return mockS3Client;
    }
    private static async Task<SelectObjectContentResponse> GetSelectObjectContentResponse(SelectObjectContentRequest request,
    CancellationToken cancellationToken = default)
    {
        var streamObj = new MemoryStream();
        // set json string based on request.Key
        var jsonString = "{a:1,b:1},{a:2,b:2}";
        await System.Text.Json.JsonSerializer.SerializeAsync(streamObj, jsonString);
        streamObj.Position = 0;
        var response = new SelectObjectContentResponse {
            Payload = new SelectObjectContentEventStream(streamObj ) {
            }
        };
        return response;
    }
}

Below is my logic

private async Task QueryS3Select<T>(string bucket, string fileKey, string expression) where T : new()
    {
        var s3client = await s3.CreateClient(bucket);
        SelectObjectContentRequest request = new SelectObjectContentRequest();
        request.BucketName = bucket;
        request.Key = fileKey;
        request.Expression = $"SELECT * FROM s3object {expression}";
        request.ExpressionType = ExpressionType.SQL;
        request.InputSerialization = new InputSerialization
        {
            Parquet = new ParquetInput()
        };
        request.OutputSerialization = new OutputSerialization
        {
            JSON = new JSONOutput
            {
                RecordDelimiter = ","
            }
        };
        SelectObjectContentResponse response = await s3client.SelectObjectContentAsync(request);
        using (var eventStream = response.Payload)
        {
            eventStream.ExceptionReceived += (sender, args) => throw args.EventStreamException;
            var recordResults = eventStream
                .Where(ev => ev is RecordsEvent)
                .Cast<RecordsEvent>()
                .Select(records =>
                {
                    using (var reader = new StreamReader(records.Payload, Encoding.UTF8))
                    {
                        return reader.ReadToEnd();
                    }
                }).ToList();
        }
    }

Edit 2: Solution. We need to mock the interface ISelectObjectContentEventStream.as below

internal class MockSelectObjectContentEventStream : ISelectObjectContentEventStream { private readonly List s3Events; public int BufferSize { get => throw new NotImplementedException(); set => throw new NotImplementedException(); }

    public event EventHandler<EventStreamEventReceivedArgs<IS3Event>> EventReceived;
    public event EventHandler<EventStreamExceptionReceivedArgs<S3EventStreamException>> ExceptionReceived;
    public event EventHandler<EventStreamEventReceivedArgs<RecordsEvent>> RecordsEventReceived;
    public event EventHandler<EventStreamEventReceivedArgs<StatsEvent>> StatsEventReceived;
    public event EventHandler<EventStreamEventReceivedArgs<ProgressEvent>> ProgressEventReceived;
    public event EventHandler<EventStreamEventReceivedArgs<ContinuationEvent>> ContinuationEventReceived;
    public event EventHandler<EventStreamEventReceivedArgs<EndEvent>> EndEventReceived;

    public MockSelectObjectContentEventStream(List<IS3Event> S3Events)
    {
        s3Events = S3Events;
    }
    public void Dispose()
    {
    }
    public IEnumerator<IS3Event> GetEnumerator()
    {
        return s3Events.GetEnumerator();
    }

    public void StartProcessing()
    {
    }

    public Task StartProcessingAsync()
    {
        return Task.CompletedTask;
    }

    IEnumerator IEnumerable.GetEnumerator()
    {
        return s3Events.GetEnumerator();
    }
}

Then the mock method can return the response as below.

private static async Task<SelectObjectContentResponse> GetSelectObjectContentResponse(SelectObjectContentRequest request,
    CancellationToken cancellationToken = default)
{
    var jsonstring = "{}";
    switch (request.Key) {
        case "request1":
            jsonstring = "{a:1}";
            break;
        case "request2":
            jsonstring = "{a:1}";
            break;
    }
    var response = new SelectObjectContentResponse
    {
        Payload = new MockSelectObjectContentEventStream(
new List<IS3Event> {
    new RecordsEvent(
        new EventStreamMessage(
            new List<IEventStreamHeader> { },Encoding.UTF8.GetBytes(jsonstring)))});
    };
    return response;
}

Answer 1

The production code is creating a new AmazonS3Client(credentials, _s3Config) client by using the new keyword:

private async Task<IAmazonS3> CreateClient(string bucketName)
{
    var credential = await _roleService.Assume(bucketName);
    var client = new AmazonS3Client(credentials, _s3Config);
    return client;
}

This means that the s3client object used by the QueryS3Select method cannot be swapped out by a Test Double. The current design is therefore not testable by means of Test Doubles.

You'll need to either find another way to test the system, or change the design of the System Under Test (SUT). This involves changing the code.

The gentlest way to make the code testable is often to introduce a Factory Method. The current code already has a good candidate for that. Making CreateClient protected virtual is a three-word edit:

protected virtual async Task<IAmazonS3> CreateClient(string bucketName)
{
    var credential = await _roleService.Assume(bucketName);
    var client = new AmazonS3Client(credentials, _s3Config);
    return client;
}

You can now create a test-specific class that inherits from the SUT. You can do that by writing a new class in your test code base, and have that new class override CreateClient to return an injected IAmazonS3 object created and configured by Moq. You then test that test-specific object, rather than the actual SUT class.

You can also use Moq to create such a SUT Double, but you'll need to use the Moq.Protected namespace to do that, and people tend to have trouble with that API. If you want to use Moq for overriding CreateClient, it'd be easier if you make that method public virtual rather than protected virtaul, in which case you can use the standard Moq API.

While using a Factory Method is the least intrusive change I can think of, I'd recommend that you consider making use of Dependency Injection as a general long-term design strategy.