CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
BaronSamedi1958
BaronSamedi1958

Reputation: 338

AWS S3 Write At Offset

is there any possibility to write at some offset inside S3 stored file? We really really don't want to download it for read-modify-write all the time because files are rather big (few GBs each).

Upvotes: 5

Views: 1239

Answers (2)

CZ workman
CZ workman

Reputation: 195

As of November 2024, AWS introduced support for appending data to objects stored in Amazon S3 Express One Zone buckets.

Example in JavaScript AWS SDK v3

s3.send(
  new PutObjectCommand({
    Bucket: 'bucket',
    Key: 'fileKey',
    Body: 'some body',
    WriteOffsetBytes: 123,
  })
)

Upvotes: 0

ketan vijayvargiya
ketan vijayvargiya

Reputation: 5659

There is no way to append data in S3.

One possible workaround could be to create new files every time (possibly using Kinesis Firehose) and run EMR jobs (possibly using Data Pipeline) to merge these small files at hourly or daily cadence as needed.

Upvotes: 4

Related Questions