CODEWITHSUNDEEP
pythonamazon-web-servicesgzipamazon-kinesis
binu456m
binu456m

Reputation: 71

can we push the compressed file (Gzip) directly into Kinesis Streams?

i have a GZIP file which need to push to kinesis as producer. i do not need records to push, all i need is push the gzip file to Stream. is it possible? i am not seeing any documentation anywhere.

Upvotes: 1

Views: 2452

Answers (1)

Marcin
Marcin

Reputation: 238259

In Kinesis Streams, the unit of data uses is called the Record. One component of the record is Data:

The data blob. The data in the blob is both opaque and immutable to Kinesis Data Streams, which does not inspect, interpret, or change the data in the blob in any way. When the data blob (the payload before base64-encoding) is added to the partition key size, the total size must not exceed the maximum record size (1 MiB). Type: Base64-encoded binary data object.

Therefore, you can push your GZIP into the stream when you encode it into base64 and the its size is less then 1MB.

If your GZIP file is larger than 1MB you have to split it into multiple chunks, and inject them as individual records. The consumer of the string would need to reconstruct the original larger file from the chunks. Alternatively, you can store it in S3, for example, and then just push a link to the object in S3 into Kinesis.

Upvotes: 2

Related Questions