CODEWITHSUNDEEP
pythonamazon-web-servicesendpointamazon-vpcamazon-kinesis-firehose
korven11
korven11

Reputation: 31

Lambda in VPC cannot connect to AWS services

I have a lambda in a VPC because it needs to access a database. Now this lambda will also have to be able to use Firehose Kinesis Video streams (https://aws.amazon.com/kinesis/video-streams/). My lambda is built in Python and this is my code for creating a kinesis video client:

client = boto3.client('kinesisvideo')
def create_stream(stream_name):
    response = client.create_stream(
        DeviceName='BE',
        StreamName=stream_name,
        MediaType='video/h264',
        DataRetentionInHours=1,
        Tags={
            'string': 'Livestream'
        }
    )
    stream_ARN = response['StreamARN']
    print('Printing ARN: ', stream_ARN)
    return stream_ARN

Now when I call create_stream('TEST'), my lambda times out after 90 seconds:

[DEBUG] 2019-11-05T15:02:47.66Z 2e094ebe-8a92-4a10-ab6c-433cf223cb5b retry needed, retryable exception caught: Connect timeout on endpoint URL: "https://kinesisvideo.eu-west-1.amazonaws.com/createStream"
Traceback (most recent call last):
File "/var/runtime/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/var/runtime/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/var/runtime/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/runtime/botocore/httpsession.py", line 262, in send
chunked=self._chunked(request.headers),
File "/var/runtime/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/var/runtime/urllib3/util/retry.py", line 344, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/var/runtime/urllib3/packages/six.py", line 686, in reraise
raise value
File "/var/runtime/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/var/runtime/urllib3/connectionpool.py", line 344, in _make_request
self._validate_conn(conn)
File "/var/runtime/urllib3/connectionpool.py", line 843, in _validate_conn
conn.connect()
File "/var/runtime/urllib3/connection.py", line 316, in connect
conn = self._new_conn()
File "/var/runtime/urllib3/connection.py", line 165, in _new_conn
(self.host, self.timeout))
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7fcf62a7e128>, 'Connection to kinesisvideo.eu-west-1.amazonaws.com timed out. (connect timeout=60)')

Of what I understand, my lambda fails to connect to https://kinesisvideo.eu-west-1.amazonaws.com/createStream because its in a VPC. To test this, I created a new temporary lambda outside my VPC and it worked perfectly fine, it could connect to Kinesis Video Stream without problems.

To fix this issue, I understand that I have to create an Endpoint so that my lambda in my VPC can access a AWS service such as Kinesis Video Streaming. I went to the VPC endpoint console and created a new endpoint. I selected the "com.amazonaws.eu-west-1.kinesis-streams" service and made sure to use the same VPC, subnets and Security group as my lambda. Once the endpoint was created, I tried calling create_stream again. Unfortunatly, I get the same result, a timeout when trying to connect.

So my question is: Is it possible at all to a AWS service from an VPC? If so, how?

UPDATE

As a response to answers I have now tried the following but I still get the same error:

My Lambda in VPC has a SG with All outbound permission. Then I created a new SG that has inbound from the VPC SG and then assigned it to the endpoint.

Upvotes: 3

Views: 2782

Answers (3)

John Rotenstein
John Rotenstein

Reputation: 269101

I just looked more closely at your code. It appears that you are using Amazon Kinesis Video, which is different to Amazon Kinesis Data Streams.

From what I can see, the VPC Endpoint is available for Data streams, but no Video.

That's why it can't reach the kinesisvideo.eu-west-1.amazonaws.com endpoint.

If so, then you'll need to launch a NAT Gateway in the public subnet, and update Route Tables accordingly. The Lambda function will then be able to reach Kinesis.

Upvotes: 0

John Rotenstein
John Rotenstein

Reputation: 269101

When an AWS Lambda function is not connected to a VPC, it has full access to the Internet.

When an AWS Lambda function is connected to a VPC, it will not have direct access to the Internet. To grant such access, you will need either:

  • A NAT Gateway in a public subnet, with matching Route tables, or
  • A VPC Endpoint in the VPC for the desired service

If using a VPC Endpoint, configure the security groups as follows:

  • Create a security group for the Lambda function (Lambda-SG)
    • Grant default "All Outbound" permission
    • There is no need for an Inbound rule
  • Create a security group for the VPC Endpoint (Endpoint-SG)
    • Grant Inbound access from Lambda-SG

That is, the Endpoint-SG specifically references Lambda-SG in the Inbound rules. This will permit the Lambda function to access the VPC Endpoint.

Upvotes: 3

error404
error404

Reputation: 2823

Yes, it is very much possible to call other services via a lambda deployed into the vpc.

As mentioned endpoint is needed which you have already configured. In this case the issue seems to be with the outbound ports.

Have you configured the outbound in the security group attached to your lambda ?

Upvotes: 1

Related Questions