CODEWITHSUNDEEP
pythonnumpyamazon-s3python-s3fs
nad
nad

Reputation: 2860

How to write a numpy array as a csv to S3

I have a numpy ndarray with 2 columns that looks like below

[[1.8238497e+03 5.2642276e-06]
 [2.7092224e+03 6.7980350e-06]
 [2.3406370e+03 6.6842499e-06]
 ...
 [1.7234612e+03 6.6842499e-06]
 [2.1071147e+03 2.1332115e-05]
 [2.6937273e+03 6.2854801e-06]]

I want to write it as a csv with column names in S3 directly (without creating a local csv file) using s3fs. I know how to do it if I had a dataframe and I can convert my numpy to dataframe. But I would like a way to do it without the need to converting the numpy array to a dataframe.

Below is the code I have

output = [[1.8238497e+03 5.2642276e-06]
     [2.7092224e+03 6.7980350e-06]
     [2.3406370e+03 6.6842499e-06]
     ...
     [1.7234612e+03 6.6842499e-06]
     [2.1071147e+03 2.1332115e-05]
     [2.6937273e+03 6.2854801e-06]]
## write the outpout in csv at s3
s3 = s3fs.S3FileSystem(anon=False)
dftowrite = pd.DataFrame({'col1': output[:, 0], 'col2': output[:, 1]})

with s3.open(f"{'s3://bucket'}/result.csv",'w') as f:
     dftowrite.to_csv(f)

I would like to do it without converting the NumPy array output to a dataframe dftowrite. Any suggestion?

Upvotes: 1

Views: 1139

Answers (1)

jellycsc
jellycsc

Reputation: 12349

Try this

output = [[1.8238497e+03 5.2642276e-06]
     [2.7092224e+03 6.7980350e-06]
     [2.3406370e+03 6.6842499e-06]
     ...
     [1.7234612e+03 6.6842499e-06]
     [2.1071147e+03 2.1332115e-05]
     [2.6937273e+03 6.2854801e-06]]
## write the outpout in csv at s3
s3 = s3fs.S3FileSystem(anon=False)

with s3.open(f"{'s3://bucket'}/result.csv",'w') as f:
    numpy.savetxt(f, output, delimiter=",")

Upvotes: 1

Related Questions