Consuming an External API that has a complex filter syntax

Question

I need to List/Get/Update/Create/Destroy (ie perform CRUD activites) on data from an external REST API.

This API has a custom filter syntax, which looks like this:

{{BaseUrl}}/V1.0/<Entity>/query?search={"filter":[{"op":"eq","field":"id","value":"68275"}]}

This filter syntax is quite flexible, and basically allows you to do fieldA == x AND/OR fieldB != y queries.

id <= 1000 && Title == "Some title"

{
    "filter": [
        {
            "op": "le",
            "field": "id",
            "value": 1000
        },
        {
            "op": "eq",
            "field": "Title",
            "value": "Some title"
        }
    ]
}

firstname == "john" || lastname != "Jones"

{
    "filter":  [
        {
            "op": "or",
            "items": [
                {
                    "op": "eq",
                    "field": "firstname",
                    "value": "John"
                },
                {
                    "op": "ne",
                    "field": "lastname",
                    "value": "Jones"
                }
            ]
        }
    ]
}  

If you're curious, it's the Autotask API: https://ww3.autotask.net/help/DeveloperHelp/Content/APIs/REST/General_Topics/REST_Swagger_UI.htm

At the moment, I have some classes which can convert to the first example query id <= 1000 && Title == "Some title".

    public interface IAutotaskFilter
    {
        string Field { get; }
        string Value { get; }
        string ComparisonOperator { get; }
    }
    
    public interface IAutotaskQuery
    {
        void AddFilter(IAutotaskFilter autotaskFilter);
        void AddFilters(IList<IAutotaskFilter> filters);
        void RemoveFilter(IAutotaskFilter autotaskFilter);
    }

The problem is that this violates the clean architecture I have.

Clean Architecture Example from Microsoft

If I use these classes (through the above interfaces), then my Application Layer will depend on an implementation detail of my Infrastructure Layer. In other words, my business logic will know how to construct queries for this specific external API.

As far as I can tell, I have a few options:

• Implement a custom LINQ provider that will translate linq queries to this syntax. From what I can tell, this is very difficult. I didn't find any recent information on this. Any libraries that have tried haven't been touched in at least 7 years. If this is possible, I would love to do it. Having that fluent syntax for queries is very appealing to me.

• Suck it up and use as is.

• Cache API entities locally and do cache invalidation using their webhooks 'updated'/'created'/'destroyed' events. That's a lot of processing, data transfer and complexity, so probably not worth it.

I am hoping there's a better option, that I haven't found. Please let me know.

If it helps, the Autotask API provides an OpenAPI v1 doc.

Answer 1

Have a look at the Expression class and ExpressionVisitor in the system.linq.expressions namespace https://learn.microsoft.com/en-us/dotnet/api/system.linq.expressions?view=net-5.0

It is the essence of LINQ, but without the SQL database part. It allows you to write expressions in C# and translate them to any format. See code example below. By assigning a lambda expression like (bo) => bo.MyID == 1 || bo.MyID == 3 to and Expression type the raw expression becomes available in code.

That is a feature of the C# compiler which LINQ uses to capture the query before it is compiled to byte code.

Change the TextWriter part of the code below to write a string in the format required by the API.

class BObject
{
    public int MyID;
}

class MyExpressionVisitor : ExpressionVisitor
{
    private TextWriter writer;

    public MyExpressionVisitor(TextWriter writer)
    {
        this.writer = writer;
    }

    protected override Expression VisitBinary(BinaryExpression node)
    {
        writer.WriteLine("Binary node " + node.NodeType);
        return base.VisitBinary(node);
    }
    protected override Expression VisitConstant(ConstantExpression node)
    {
        writer.WriteLine("Constant node " + node.NodeType + " Value " + node.Value);
        return base.VisitConstant(node);
    }

    protected override Expression VisitMember(MemberExpression node)
    {
        writer.WriteLine("Constant node " + node.NodeType + " Member " + node.Member);
        return base.VisitMember(node);
    }
}

class Program
{
    static void Main(string[] args)
    {
        Expression<Func<BObject, bool>> query = (bo) => bo.MyID == 1 || bo.MyID == 3;
        var visitor = new MyExpressionVisitor(Console.Out);

        visitor.Visit(query.Body);

        Console.ReadKey();
    }
}

Output:

Binary node OrElse
Binary node Equal
Constant node MemberAccess Member Int32 MyID
Constant node Constant Value 1
Binary node Equal
Constant node MemberAccess Member Int32 MyID
Constant node Constant Value 3

Answer 2

I've though a bit on what can be done to not introduce new models in your current Domain. And came up with a solution, that adds new dependencies, but doesn't require new Domain models. Here is a possible solution that can be used:

An abstraction, that will be providing you your domain entities based on business needs:

 public interface IEntityProvider
 {
     Task<Entity> GetEntityAsync(string structuredData);
 }

"structuredData" will be a string, that will be helping you construct your filters to make 3rd party request. To get it, you will have a helper, that will be doing the job for you:

    public interface IRequestEncoder
    {
        string GetStructuredData(/* arguments */);
    }

    public class RequestEncoder : IRequestEncoder
    {
        public static string GetStructuredData(/* arguments */)
        {
            // Structuring the data in a way, that can be later mapped 1 to 1 to filters.
        }
    }

Also you will need a decoder, that will be injected into the implementation of the "EntityProvider":

   public interface IRequestDecoder
   {
       IEnumerable<IAutotaskFilter> GetFilter(string structuredData);
   }

And finally, the encapsulated module with the filters logic may look something like this: (It will be sitting outside the domain along with your filter models)

    public class EntityProvider : IEntityProvider
    {
        private readonly IRequestDecoder _requestDecoder;
        public EntityProvider(IRequestDecoder requestDecoder)
        {
            _requestDecoder = requestDecoder;
        }
        public async Task<Land> GetEntityAsync(string structuredData)
        {
            // Analyze the structure data, make sure it's decodable
            // Convert structureData to filters using corresponding component (IRequestDecoder)
            // Request the external service, passing necessary parameters
            // Return domain model
        }
    }

P.S. Of course interfaces/contracts may, and probably do, differ from your business needs reality, also you may come up with better names of the components, but I hope the idea gives you some value to build upon.