CODEWITHSUNDEEP
pythonarrays
Noor
Noor

Reputation: 365

How to save a large array in txt file in python

I have 2D array: 

import numpy as np

output = np.array([1,1,6])*np.arange(6)[:,None]+1

output
Out[32]: 
array([[ 1,  1,  1],
       [ 2,  2,  7],
       [ 3,  3, 13],
       [ 4,  4, 19],
       [ 5,  5, 25],
       [ 6,  6, 31]])

I tried to use np.savetxt('file1.txt', output, fmt='%10d') i have got the result in one line only

How can I save it in txt file simillar to :

        x   y    z 
        1   1    1
        2   2    7
        3   3   13
        4   4   19
        5   5   25
        6   6   31

3 separate columns, each column has name (x,y,z)

Please note: the original array too large (40000000 rows and 3 columns), I am using Python 3.6 I have tried the solutions in here and here but, it does not work with me

Upvotes: 0

Views: 5593

Answers (1)

Patrick Artner
Patrick Artner

Reputation: 51643

Noor, let me guess - you are using windows notepad to view the file?

I use Notepad++ which is smart enough to understand Unix-style-Lineendings which are used (by default) when creating files by np.savetxt() even when operated under windows.

You might want to explicitly specify newline="\r\n" when calling savetxt.

np.savetxt('file1.txt', output, fmt='%10d' ,header= "       x          y          z", newline="\r\n")

Doku: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.savetxt.html

I am not sure about your data, but this:

import numpy as np

output = np.array([1,1,6])*np.arange(60)[:,None]+1

print(output)


np.savetxt('file1.txt', output, fmt='%10d' ,header= "       x          y          z")

Produces this output:

#        x          y          z
         1          1          1
         2          2          7
         3          3         13 
       === snipped a few lines ===
        58         58        343
        59         59        349
        60         60        355

for me.

  • for np.arange(1000000) its about 32MB big and similarly formatted...

  • for np.arange(10000000) its about 322MB big and similarly formatted...

willem-van-onsem 1+Gb was far closer.

I did not account for the spacing of fixed 10 chars per number, my bad.

Upvotes: 2

Related Questions