io-ts logging decoding errors for union types in arrays

Question

I am using io-ts and trying to decode nested arrays of values. The default io-ts behavior is that if any item in an array fails, it fails the whole array. I still wanted the array to pass and only return the valid items and be able to log the invalid ones. I wrote a custom codec that extends the array using withValidate of io-ts-types.

export function validItemsOnlyArray<C extends t.Mixed>(theType: C) {
  return withValidate(t.array(theType), (itemToDecode, context) => {
    const pipeResult = pipe(
      // validate that the value is at least an array
      // if it passes, then chain and validate each item
      t.UnknownArray.validate(itemToDecode, context),
      FpTsEither.chain((validArrayObject) => {
        const decoded = pipe(
          validArrayObject,
          FpTsArray.map((arrayItem) => {
            const decodeResult = theType.decode(arrayItem)
            if (FpTsEither.isLeft(decodeResult))
              console.log(
                `${getPaths(decodeResult)} unable able to decode ${
                  theType.name
                }`,
              )
            //TODO: figure out how to log errors, currently logging for each failed union type
            return isRight(decodeResult)
              ? t.success(decodeResult.right)
              : t.failure(arrayItem, context)
          }),
          FpTsArray.separate,
        )

        return t.success(decoded.right)
      }),
    )
    return pipeResult
  })
}

The decoding works perfectly. However, I get false positives in my logging statements because I am using union types. If one of the union fails, it tries the next one until it succeeds. This returns the correct output, however, it also logs the failures when a union type fails to decode.

I need some way to know that we are trying to decode a union type and there are still other objects to try before we log that all possible options have been tried. I know there is a lot of information available in context but wasn't sure if there was a better way to know and stitch it all together.

Here are the objects I'm trying to decode

const Competitor = t.exact(
      t.intersection([
        t.type({
          isWinner: t.boolean,
          name: t.string,
          shortName: t.string,
        }),
        t.partial({
          gender: t.union([t.string, t.null]),
          dateOfBirth: t.union([t.string, t.null]),
          zipCode: t.union([t.string, t.null]),
          city: t.union([t.string, t.null]),
          state: t.union([t.string, t.null]),
        }),
      ]),
    )

    const SportResult = t.exact(
      t.intersection([
        t.type({
          arenaBoutGuid: t.string,
          arenaEventGuid: t.string,
          competitors: validItemsOnlyArray(Competitor),
          isDual: t.boolean,
          dateModified: t.string,
          eventName: t.string,
          sport: t.string,
          siteId: t.number,
          type: t.string,
        }),
        t.partial({
          boutNumber: t.union([t.string, t.null]),
          winType: t.union([t.string, t.null]),
        }),
      ]),
    )

    const Comp = t.intersection([
      t.type({
        name: t.string,
        id: t.string,
        isWinner: t.boolean,
        points: t.union([t.number, t.null]),
      }),
      t.partial({
        shortName: t.union([t.string, t.null]),
        coreNodeId: t.union([t.number, t.null]),
      }),
    ])

    const Sport2Result = t.exact(
      t.intersection([
        t.type({
          arenaBoutGuid: t.string,
          arenaEventGuid: t.string,
          competitors: validItemsOnlyArray(Comp),
        }),
        t.partial({
          winType: t.union([t.string, t.null]),
          matchScore: t.union([t.string, t.null]),
          weightClassId: t.union([t.string, t.null]),
          weightClassName: t.union([t.string, t.null]),
        }),
      ]),
    )

When decoding here, result decodes correctly as Right, but I get the console.log message that it failed.

    const Response = t.type({
      data: validItemsOnlyArray(t.union([Sport2Result, SportResult])),
    })

   const result = Response.decode({
  data: [
    {
      arenaBoutGuid: 'xyz',
      arenaEventGuid: 'abvc',
      competitors: [
        {
          isWinner: true,
          name: 'M Name',
          shortName: 'M. Name',
        },
        {
          isWinner: false,
          name: 'A Name',
          shortName: 'A. Name',
        },
      ],
      siteId: 1,
      dateModified: '2022-05-06T22:03:17.060Z',
      matchDateTime: '2022-05-06T22:02:01.417Z',
      isDual: false,
      ventName: 'Event',
      sport: 'my sport',
      matchScore: '10-0 3:03',
      type: 'match',
    },
  ],
})

Any help would be appreciated!

Answer 1

The only answer I can come up with that accomplishes what you're hoping for and doesn't require writing your own custom wrapper around io-ts to add state to your decoding, is to forgo logging in the custom codec that separates out the failures.

The problem boils down to the fact that each value will potentially be validated multiple times when using t.union so the logging cannot happen within the validation function and must be delayed to a later point.

I think the simplest approach would be to change the return type of your custom codec to return Separated<t.Errors[], A[]>. This means that in you code when using the decoded value, you will need to look at field.right, and the error information will be carried around with you, but there doesn't seem to be another way to handle the different possible ways of interpreting the left values when union fields are involved.

With minimal changes to your code this would look like:

// Previous imports +
import { flow } from 'fp-ts/lib/function';

interface IFailure {
  typeName: string;
  failure: t.Errors;
}

export function validItemsOnlyArray<C extends t.Mixed>(theType: C) {
  return withValidate(t.array(theType), (itemToDecode, context) => {
    const pipeResult = pipe(
      // validate that the value is at least an array
      // if it passes, then chain and validate each item
      t.UnknownArray.validate(itemToDecode, context),
      FpTsEither.map(flow(
          FpTsArray.map(theType.decode),
          FpTsArray.separate,
          // Adding in a bit of context so the error can be crafted later.
          FpTsSeparated.mapLeft((l): IFailure[] => l.map(f => ({
            failure: f,
            typeName: theType.name,
          }))),
          FpTsEither.right, // Or t.success
        )
      ),
    )
    return pipeResult
  })
}

In practice, you will need to look through the successfully parsed values to search for items that were not successfully parsed by any of the branches in the union and then add logging just for those items. That logic might be better contained in a helper function, rather than as part of your codecs anyway.

You could simplify working with this slightly with a helper like:

function getOrLog<A>(s: Separated<IFailure[], A[]>): A[] {
  s.left.forEach((e) => {
    console.log(
      `${getPaths(e.failure)} unable able to decode ${e.typeName}`
    );
  });
  return s.right;
}

This would at least get you back to where you are now, when you want to use one of the fields, but you could also just use the field's right value directly.

In the union case, as I mentioned, you will likely need to look through the left values in an aggregation function of some time to find values that did not succeed anywhere.

Alternative Approach

The other option that presents itself is to customize union as you mentioned. I think this is going to be a more complicated endeavor, but it would be possible to create a parallel abstraction that internally uses io-ts but that adds state to the decode process. This would allow your custom codec to write directly into that stream and you could define a custom union helper to manage the stream when it's children use it.

While this would offer more flexibility, it comes at a significant cost in needing to build out a mirror of io-ts for each of the edge cases you want to support, which is why I ultimately didn't spend much time considering it.