CODEWITHSUNDEEP
jsonamazon-s3aws-cli
崔起鎮
崔起鎮

Reputation: 1

json files downloaded by aws s3 sync / aws s3 cp are incomplete

I downloaded .json file from amazon s3, but its content is just a value of the first key/value pair.

original json file is like this:

{
    "_1": [
        {
            "Name": "name",
            "Type": "type"
        }
    ]
}

but downloaded json file is not even json file, it only has list inside.

[
  {
    "Name": "name",
    "Type": "type"
  }
]

I tried aws s3 sync / aws s3 copy / aws s3api get-object and all of its result is same.

I only want to download original file from the s3 bucket.

is there any solution?

I just copied the original content on the s3 select from preview and saved it as a file.

I found out its md5 checksum and file size is totally different with the object overview.

It seems the original file on the s3 bucket is corrupted, but I'm not sure how its preview is still same as the original content.

Upvotes: 0

Views: 885

Answers (1)

崔起鎮
崔起鎮

Reputation: 1

I found out aws s3api select-object-content can give me same result as the S3 select from preview but without indents.

For indents, I decided to reindent after I receive the uncorrupted results.

I used below command to retrieve my json files.

aws s3api select-object-content \
  --bucket $BUCKET \
  --key $KEY --expression "select * from s3object" \
  --expression-type 'SQL' \
  --input-serialization '{"JSON": {"Type": "LINES"}, "CompressionType": "NONE"}' \
  --output-serialization '{"JSON": {}}' \
  /dev/stdout | python -mjson.tool > $KEY

Upvotes: 0

Related Questions