Approach to testing with FsCheck

Question

I am trying to make the paradigm shift to FsCheck and random property-based testing. I have complex business workflows that have more test cases than I can possibly enumerate, and the business logic is a moving target with new features being added.

Background: Match-making is a very common abstraction in Enterprise Resource Planning (ERP) systems. Order fulfillment, supply chain logistics, etc.

Example: Given a C and a P, determine if the two are a Match. At any given point in time, some Ps are never Match-able, and some Cs are never Match-able. Each has a Status that says whether they can be even considered for a Match.

public enum ObjectType {
  C = 0,
  P = 1
}

public enum CheckType {
  CertA = 0,
  CertB = 1
}
public class Check {
  public CheckType CheckType {get; set;}
  public ObjectType ObjectType {get; set;}
  /* If ObjectType == CrossReferenceObjectType, then it is assumed to be self-referential and there is no "matching" required. */
  public ObjectType CrossReferenceObjectType {get; set;}
  public int ObjectId {get; set;}
  public MatchStatus MustBeMetToAdvanceToStatus {get; set;}
  public bool IsMet {get; set;}
}

public class CStatus {
  public int Id {get; set;}
  public string Name {get; set;}
  public bool IsMatchable {get; set;}
}

public class C {
  public int Id {get; set;}
  public string FirstName {get; set;}
  public string LastName {get; set;}
  public virtual CStatus Status {get;set;}
  public virtual IEnumerable<Check> Checks {get; set;}
  C() {
    this.Checks = new HashSet<Check>();
  }
}

public class PStatus {
  public int Id {get; set;}
  public string Name {get; set;}
  public bool IsMatchable {get; set;}
}

public class P {
  public int Id {get; set;}
  public string Title {get; set;}
  public virtual PStatus Status { get; set;}
  public virtual IEnumerable<Check> Checks {get; set;}
  P() {
    this.Checks = new HashSet<Check>();
  }
}

public enum MatchStatus {
  Initial = 0,
  Step2 = 1,
  Step3 = 2,
  Final = 3,
  Rejected = 4
}

public class Match {
  public int Id {get; set;}
  public MatchStatus Status {get; set;}
  public virtual C C {get; set;}
  public virtual P P {get; set;}
}

public class MatchCreationRequest {
  public C C {get; set;}
  public P P {get; set;}
}

public class MatchAdvanceRequest {
  public Match Match {get; set;}
  public MatchStatus StatusToAdvanceTo {get; set;}
}

public class Result<TIn, TOut> {
  public bool Successful {get; set;}
  public List<string> Messages {get; set;}
  public TIn InValue {get; set;}
  public TOut OutValue {get; set;}
  public static Result<TIn, TOut> Failed<TIn>(TIn value, string message)
  {
    return Result<TIn, TOut>() {
      InValue = value,
      Messages = new List<string>() { message },
      OutValue = null,
      Successful = false
    };
  }
  public Result<TIn, TOut> Succeeded<TIn, TOut>(TIn input, TOut output, string message)
  {
    return Result<TIn, TOut>() {
      InValue = input,
      Messages = new List<string>() { message },
      OutValue = output,
      Successful = true
    };
  }
}

public class MatchService {
   public Result<MatchCreationRequest> CreateMatch(MatchCreationRequest request) {
     if (!request.C.Status.IsMatchable) {
       return Result<MatchCreationRequest, Match>.Failed(request, "C is not matchable because of its status.");
     }
     else if (!request.P.Status.IsMatchable) {
       return Result<MatchCreationRequest, Match>.Failed(request, "P is not matchable because of its status.");
     }
     else if (request.C.Checks.Any(ccs => cs.ObjectType == ObjectType.C && !ccs.IsMet) {
       return Result<MatchCreationRequest, Match>.Failed(request, "C is not matchable because its own Checks are not met.");
     } else if (request.P.Checks.Any(pcs => pcs.ObjectType == ObjectType.P && !pcs.IsMet) {
       return Result<MatchCreationRequest, Match>.Failed(request, "P is not matchable because its own Checks are not met.");
     }
     else if (request.P.Checks.Any(pcs => pcs.ObjectType == ObjectType.C && C.Checks.Any(ccs => !ccs.IsMet && ccs.CheckType == pcs.CheckType))) {
       return Result<MatchCreationRequest, Match>.Failed(request, "P's Checks are not satisfied by C's Checks.");
     }
     else {
       var newMatch = new Match() { C = c, P = p, Status = MatchStatus.Initial }
       return Result<MatchCreationRequest, Match>.Succeeded(request, newMatch, "C and P passed all Checks.");
     }
   }
}

Bonus: Beyond a naive "block Match" status, C and P each has a set of Checks. Some Checks must be true for the C being Match-ed, some Checks must be true for the P being Match-ed, and some Checks for C must be cross-checked against the Checks for P. This is where I suspect model-based testing with FsCheck will pay huge dividends, since (a) it is an example of a new feature added to the product (b) I can potentially write tests (user interactions) such as:

1. Create
2. After Create, Move forward through the pipeline
3. Move backward (when is it allowed vs. not? ex: a Paid Order probably can't move back to a Purchase Approval step)
4. Add/remove stuff (like Checks) while in the middle of pipeline
5. If I ask to create a Match for the same C and P twice (e.g., concurrently with PLINQ), will I create duplicates? (What message gets returned to the user?)

Things I am struggling with:

1. How should I generate test data for FsCheck? I think the right way is to define all discrete possible combinations of Cs and Ps for creating a Match and have those be the "pre-conditions" for my model-based tests and the post-conditions be whether a Match should be created, but...
2. Is that really the right approach? It feels a too deterministic for a randomized property-based testing tool. Is it over-engineering to even use FsCheck in such a situation? Then, it's almost as if I have a data generator that ignores the seed value and returns a deterministic stream of test data.
3. At this point, is FsCheck generators any different from just using xUnit.net and something like AutoPOCO?

Answer 1

If you want to generate deterministic (including exhaustive) test data then FsCheck is not really a good fit. One of the base assumptions is that your state space is too big for that to be feasible, so random, but guided generation is able to find more bugs (it's hard to prove this, but there is definitely some research that corroborates this assumption. That is not to say it's the best approach in all circumstances).

I'm assuming from what you've wrote that the CreateMatch method is what you want to test properties of; so in that case you should try to generate a MatchCreationRequest. Since generators compose, this is in your case fairly lengthy (because they're all mutable types, there is no reflection-based automatic generator) but also easy - it's always the same pattern:

var genCStatus = from id in Arb.Generate<int>()
                 from name in Arb.Generate<string>()
                 from isMatchable in Arb.Generate<bool>()
                 select new CStatus { Id = id, Name = name, IsMatchable = isMatchable };

var genC = from status in genCStatus
           ...
           select new C { ... }

Once you have those, writing properties to test should be relatively straightforward, although in this example at least they aren't significantly simpler than the implementation itself.

One example is:

//check that if C or P are not matchable, the result is failed.
Prop.ForAll(genC.ToArbitrary(), genP.ToArbitrary(), (c, p) => {
    var result = MatchService.CreateMatch(new MatchCreationRequest(c, p));
    if (!c.IsMatchable || !p.IsMatchable) { Assert.IsFalse(result.Succesful); }
}).QuickCheckThrowOnFailure();