CODEWITHSUNDEEP
pythonjsonnumpyamazon-s3
mtngld
mtngld

Reputation: 577

Read npy file directly from S3 StreamingBody

I have many .npy (numpy) and .json files saved in s3, I need to download and load these files.

For the json files I am downloading the files and loading into dictionary using the following code:

obj = s3.Object(bucket, key)
response = obj.get()
body = response['Body'].read().decode('utf-8')
data = json.loads(body)

That way I avoid writing the .json file to disk and reading it.

Now I want to do the same for the .npy files, what would be the equivalent?

Currently I am downloading the file, saving to temp file on disk and loading with np.load but would like to avoid that if possible.

Upvotes: 0

Views: 2894

Answers (2)

matkal93
matkal93

Reputation: 9

The correct answear should be:

numpy_image = np.fromstring(s3_object.get()['Body'].read(), np.uint8)

and if you would like to get the image from it you can use cv2 like in the example below:

image = cv2.imdecode(numpy_image, cv2.IMREAD_COLOR)

Upvotes: 0

mtngld
mtngld

Reputation: 577

Following https://stackoverflow.com/a/28196540/3509999:

obj = s3.Object(bucket, key)
with io.BytesIO(obj.get()["Body"].read()) as f:
    # rewind the file
    f.seek(0)
    arr = np.load(f)

Upvotes: 3

Related Questions