AWS Lambda nodejs long running process

Question

I'm working on a project (Serverless, Lambda, Nodejs, MongoDB, SQS) where I need to make a price chart, there is an API /api/products?country=countryCode that returns data about the product and the price (see response sample below), prices can be different for each country, for example, for NL the price can be $12.99, and for AU €13.99 (all other properties do not change)

I have a collection of countries in MongoDb, there are about 225+ countries in the collection. I have a cron job that triggers a lambda function once daily. For each country, I need to call the API that returns the product data (see sample response below) that needs to be processed. After fetching all the data, I insert products and prices at the same time (products are unique, price count = product count * country count * days). To avoid price differences between countries, you need to insert the data at once. For example yesterday the price of the Sword of Bladur was $17.99, today it is $18.99, We have updated the price for NL, but AU is in process.

Please help me solve one of these problems:

• Problem 1 (Cron -> Handler): 1 API request takes about 6-8 seconds. To process all countries, it takes ~30 (225 * 8 / 60) minutes, however, there is one small problem, lambda function has timeout limit (max: 15 minutes), of course this time is not enough to finish the job.

• Problem 2 (Cron -> Handler -> SQS Handler): I rewrote my code and instead of running one lambda function, I send each country to AWS SQS (Consumer/Producer), which triggers a lambda function that fetches data from API and processes, however, here I have a problem with the fact that I do not know when the cycle will end to insert prices at once.

P.S. In my opinion I should use Producer/Consumer to avoid losing country data, so I will be glad if someone has solutions for the second problem

MongoDB Product model
{
  id: String,
  title: String,
  ...
}

MongoDB Price model
{
  country: String,
  productId: String,
  price: Number
}

API response sample:

GET /api/products?country=NL
[
  {
    "id": "37071265-7f98-4e32-ae45-c23f83e7c7a2",
    "title": "Dusty Book",
    "country": "NL",
    "price": 1299,
    ...
  },
  {
    "id": "49701bcf-c076-4064-b331-0952aee21deb",
    "title": "Sword of Bladur",
    "country": "NL",
    "price": 1799,
    ...
  },
  ...
]
-----
GET /api/products?country=AU
[
  {
    "id": "37071265-7f98-4e32-ae45-c23f83e7c7a2",
    "title": "Dusty Book",
    "country": "AU",
    "price": 1549,
    ...
  },
  {
    "id": "49701bcf-c076-4064-b331-0952aee21deb",
    "title": "Sword of Bladur",
    "country": "AU",
    "price": 1799,
    ...
  },
  ...
]

Answer 1

Have you considered using Step Functions? You could one a single function that gets a list of the countries. The results of that could be passed to a map state that would call a lambda for each country, getting the required data. Once all of those functions are done it can call another lambda that would perform the final updates in the database.