CODEWITHSUNDEEP
karate
steve1337
steve1337

Reputation: 147

Test async data processing flows with Karate Labs

I'm looking for best practices or the recommended approach to test async code execution with Karate.

Our use cases are all pretty similar but a basic one is:

  1. Client makes HTTP request to API
  2. API accepts request and creates a messages which is added to a queue
  3. API replies with ACCEPTED / 202
  4. Worker picks up message from queue processes it and updates database
  5. Eventually after the work is finished another endpoint delivers updated data

How can I check with Karate that after processing has finished other endpoints return the correct result?

Concrete real life example:

  1. Client requests a processing intensive data export to API e.g. via HTTP POST /api/export
  2. API create message with information for creating the export and puts it on AWS SQS queue
  3. API replies with 202
  4. Worker receives message and creates export, uploads result as ZIP to S3 and finally creates and entry in the database symbolizing this export
  5. Client can now query list exports endpoint e.g. via HTTP GET /api/exports
  6. API returns 200 with the list of exports incl. the newly created entry

Generally I have two ideas on how to approach this:

  1. Use karate retry until on the endpoint that returns the list of exports
  2. In the API response (step #3) return the message ID and use the HTTP API of SQS to poll that endpoint until the message has been processed and then query the list endpoint to check the result

Is either of those approach recommended or should I choose an entirely different solution?

Upvotes: 1

Views: 573

Answers (1)

Peter Thomas
Peter Thomas

Reputation: 58088

The moment queuing comes into the picture, I would not recommend retry until. It would work if you are in a hurry, but if you are ok to write a little bit of Java code, please read on. Note that this Java "glue code" needs to be written only once, and then the team responsible for writing the functional flows will be up and running.

I personally would prefer option (2) just because when a test fails, you will have a lot more diagnostic information and traces to look at.

Pretty sure you won't have a problem using AWS Java libs to do things such as polling SQS.

I think this example will answer all your questions: https://twitter.com/getkarate/status/1417023536082812935

Upvotes: 1

Related Questions