CODEWITHSUNDEEP
amazon-web-servicesamazon-swf
jayjyli
jayjyli

Reputation: 821

SWF Activity is not completing even though the computation has finished

I'm testing a new SWF workflow, and I've got some activity that makes a RESTful call out to another service. Problem is, I can see through logging that the actual call takes less than a second to complete, but the Activity always times out in SWF (START_TO_CLOSE of 5 mins). Being more specific, the RESTful call is a list call, and when I limit the batch size to a small number, the Activity completes and moves on very quickly. But at some seemingly arbitrary threshold, it chokes completely.

Does anyone have any insight into this? I've read that SWF calls have a size limitation of 1 MB, does anyone know how to find the size of data my workers are trying to pass SWF?

Upvotes: 1

Views: 477

Answers (2)

Maxim Fateev
Maxim Fateev

Reputation: 6870

I wouldn't recommend using activity input and output parameters for passing large data sets. SWF is an orchestration technology, not the data passing one. The standard workarounds are:

  • Storing result in a separate store (S3 for example) and passing reference to it.
  • Caching result locally on a machine and route all following activities to the same host for them to have access to the cached result. See fileprocessing sample for the details of routing approach.

BTW. Have you checked out Cadence which is an open source version of SWF with much better client side libraries?

Upvotes: 1

jayjyli
jayjyli

Reputation: 821

After some remote debugging, it turns out the response from the task is too big and the activity is failing silently. The failure occurs when the framework tries to report the response back to SWF, and the SDK calls RespondActivityTaskCompleted. That API has a length restriction on the internal result param:

Length Constraints: Maximum length of 32768.

This is a validation error that throws an uncaught exception and is swallowed internally until the Activity times out.

Upvotes: 1

Related Questions