user606521
user606521

Reputation: 15474

Redis commands on AWS from ECS hang forever

I have ECS container managed with Fargate and ElastiCache Redis instance. Both ECS and Redis are deployed in private VPC. What I want is to connect to Redis from ECS container. All subnets and security groups seem to be configured correctly, after sending auth command there is "ready" event, however all other commands (info for example) hang forever. Here is code (Node.js with ioredis) which is run in container:

const client = new IoRedis(process.env.REDIS_URL, {
  connectTimeout: 5000,
  enableOfflineQueue: false,
  enableReadyCheck: false,
});

setInterval(() => {
  client
    .info()
    .then(info => {
      console.log('info received', info);
    })
    .catch(err => {
      console.error('error received', err);
    });
}, 5000);

Log output is:

12:14:00
{"name":"app","hostname":"ip-10-0-103-126.us-east-2.compute.internal","pid":1,"appName":"api","level":30,"msg":"Server listening on 5000","time":"2019-04-03T12:14:00.176Z","v":0}

12:14:00
2019-04-03T12:14:00.181Z ioredis:redis status[master.ab-cache.hp48ph.use2.cache.amazonaws.com:6379]: [empty] -> connecting

12:14:00
2019-04-03T12:14:00.263Z ioredis:redis status[10.0.31.100:6379]: connecting -> connect

12:14:00
2019-04-03T12:14:00.264Z ioredis:redis write command[10.0.31.100:6379]: 0 -> auth([ '**************************************************' ])

12:14:00
2019-04-03T12:14:00.265Z ioredis:redis status[10.0.31.100:6379]: connect -> ready

12:14:05
2019-04-03T12:14:05.185Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:14:10
2019-04-03T12:14:10.184Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:14:15
2019-04-03T12:14:15.190Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:14:20
2019-04-03T12:14:20.194Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:14:25
2019-04-03T12:14:25.195Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:14:30
2019-04-03T12:14:30.197Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:14:35
2019-04-03T12:14:35.200Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:14:40
2019-04-03T12:14:40.200Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:14:45
2019-04-03T12:14:45.202Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:14:50
2019-04-03T12:14:50.202Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:14:55
2019-04-03T12:14:55.203Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:15:00
2019-04-03T12:15:00.204Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:15:05
2019-04-03T12:15:05.205Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:15:10
2019-04-03T12:15:10.204Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:15:15
2019-04-03T12:15:15.205Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:15:20
2019-04-03T12:15:20.206Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:15:25
2019-04-03T12:15:25.206Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:15:30
2019-04-03T12:15:30.208Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:15:35
2019-04-03T12:15:35.207Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:15:40
2019-04-03T12:15:40.207Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

12:15:45
2019-04-03T12:15:45.207Z ioredis:redis write command[10.0.31.100:6379]: 0 -> info([])

Lines above come from IoRedis debug info. It seems it connects and authenticates correctly, but further commands hang forever, never returning response (neither success nor error). What might be wrong here?


Also tested it with "redis" packaged (instead of "ioredis"):

console.log('creating redis client'); // tslint:disable-line
    const redisClient = redis.createClient(redisConfig.url, {
      connect_timeout: 5000,
      enable_offline_queue: false,
      no_ready_check: true,
    });

    redisClient.on('error', err => {
      console.error('redis error', err); // tslint:disable-line
    });

    setInterval(() => {
      console.log('sending redis command'); // tslint:disable-line
      redisClient.info((err, result) => {
        console.log('redis response', { err, result }); // tslint:disable-line
      });
    }, 10000);

Same issue - command callback is never called:

enter image description here

Upvotes: 3

Views: 2625

Answers (1)

user606521
user606521

Reputation: 15474

It was caused by ElastiCache param "transit_encryption_enabled". It requires tunnel to connect to redis. After turning this param off everything works as expected.

https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/in-transit-encryption.html

Upvotes: 2

Related Questions