CODEWITHSUNDEEP
amazon-web-servicesamazon-kinesisamazon-kinesis-firehose
Christopher
Christopher

Reputation: 643

How do I log IP, timestamp, and user-agent on Kinesis server side

I've got a kinesis datastream hooked up to a kinesis firehose which dumps data in S3. I'm using the javascript client side library aws-sdk to push records into the data stream (e.g. with putRecords()).

Is there a way for me to log the ip, timestamp, and user-agent op the server side? More generally, can I put in http headers on the server side into the record which is being ingested into kinesis? Or do I have to put all of this on the client side?

(If on the client side, any pointers to best practices when dealing with ip and timestamps from clients, which might be unreliable?)

Upvotes: 5

Views: 827

Answers (1)

devonlazarus
devonlazarus

Reputation: 1307

Is there a way for me to log the ip, timestamp, and user-agent op the server side?

Unfortunately, there is no way for you to do this at the server-side. The Kinesis API does not offer any sort of configuration along these lines, and the Kinesis record that gets passed around does not contain any of the HTTP metadata that came over with the putRecords() request. (Looks something like this.) You would have to add the information at the client-side, as you surmised.

We have a similar situation where we cannot rely on the client-side details (particularly around time) so we have a small web service that sits in front of Kinesis, accepts HTTP from our clients, adds the HTTP metadata and a server-side timestamp, then forwards the record to Kinesis.

You could do something similar, or, if you think it easier, use API Gateway/Lambda to accept the data from your client, add the HTTP metadata and forward that to Kinesis inside a Lambda that backed the API Gateway endpoint.

Upvotes: 3

Related Questions