AWS ELB - ECS - RDS stack latency

Question

This is a bit of share and compare kind of question.

I have the following stack deployed on AWS: ELB > ECS Fargate > node/express > RDS

I'm (negatively) surprised by some of the latencies observed for some simple requests that involve or not DB queries:

• simple requests to /healthcheck would average at 150/200ms
• simple SELECT queries done directly to my RDS instance thru pgadmin would average at 400ms (I only have a few entries on the requested table).

I tried to search for benchmark results but couldn't find anything useful. So I'd be grateful for anyone sharing his experience for a similar stack.

Thanks a lot!

---

Additional info on the deployment:

• both ECS and RDS deployed within the same region (eu-west-1)
• requests made from Spain (could that be it?)
• ECS sits on 256 cpu units and 512 reserved memory
• I'm the only one making requests on a dev environment (is there any "cold start" on ELB?)
• RDS sits on an db.t2.micro instance and a postgresql v12.4 engine
• Thanks @Maurice, I've added the info in the ticket but here a summary:
• no utilization issue: single digit CPU utilization and memory at c. 25%; CPU never goes up 10% with several requests and memory always stable.
• I instantiate the DB connection via Sequelize when creating the app and reuse it for each request. DB pooling used via Sequelize with 4 max connections

A typical cURL latency analysis on the ELB dns:

❯ curl -kso /dev/null http://be-api-main-elb-uat.wantedtv.com -w "==============\n\n
| dnslookup: %{time_namelookup}\n
| connect: %{time_connect}\n
| appconnect: %{time_appconnect}\n
| pretransfer: %{time_pretransfer}\n
| starttransfer: %{time_starttransfer}\n
| total: %{time_total}\n
| size: %{size_download}\n
| HTTPCode=%{http_code}\n\n"
==============

| dnslookup: 0,003741
| connect: 0,065718
| appconnect: 0,000000
| pretransfer: 0,065813
| starttransfer: 0,155532
| total: 0,155639
| size: 92
| HTTPCode=200

Answer 1

ECS sits on 256 cpu units and 512 reserved memory

It might be worth allocating more resources to see if that has some improvement, especially since there are some weird hidden limitations tied to the different CPU and Memory levels that might not be that apparent at first. Since 0.25 vCPU doesn't even give you a full thread to work with there could be other preempting going on that isn't visible to you.

Outside of that there are other things you can look for-

• Is your application pooling requests to RDS, or creating new ones each time? I know you're intending to use pooling but it might be worth confirming it is actually working.
• Are you exposing your container directly to the load balancer, or using a sidecar container such as NGINX to handle request buffering?
• What happens if you hit the containers directly instead of through the load balancer? This can at least help isolate whether the issue is on the load balancer or on the container side.
• How does your application handle concurrency?
• How much data is being sent in each request? It's possible that large amounts of data may be locking up threads or processes and making other requests slow down as a result.
• Are there any other services involved that aren't obvious? I once had a service crash because the logging service we were using broke, causing the messages to queue up and lock down the services.

The basic idea with a lot of this is to try and isolate the various components to identify the one causing the slow down. I do believe it'll end up being something in the task itself (service container, sidecar, or service) considering you mentioned quick responses from the database server itself.