CODEWITHSUNDEEP
pythonnumpy
user6652926
user6652926

Reputation:

How to put many numpy files in one big numpy file, file by file?

I have 166600 numpy files, I want to put them into one numpy file: file by file, I mean that the creation of my new big file must from the begin: the first file must be read and written in the file, so the big file contains only the first file, after that I need to read and write the second file, so the big file contains the first two files.

import matplotlib.pyplot as plt 
import numpy as np
import glob
import os, sys
fpath ="path_Of_my_final_Big_File"
npyfilespath ="path_of_my_numpy_files"   
os.chdir(npyfilespath)
npfiles= glob.glob("*.npy")
npfiles.sort()
all_arrays = np.zeros((166601,8000))
for i,npfile in enumerate(npfiles):
    all_arrays[i]=np.load(os.path.join(npyfilespath, npfile))
np.save(fpath, all_arrays)

Upvotes: 3

Views: 3015

Answers (1)

languitar
languitar

Reputation: 6784

If I understand your questions correctly, you can use numpy.concatenate for this:

import matplotlib.pyplot as plt 
import numpy as np
import glob
import os, sys
fpath ="path_Of_my_final_Big_File"
npyfilespath ="path_of_my_numpy_files"   
os.chdir(npyfilespath)
npfiles= glob.glob("*.npy")
npfiles.sort()
all_arrays = []
for i, npfile in enumerate(npfiles):
    all_arrays.append(np.load(os.path.join(npyfilespath, npfile)))
np.save(fpath, np.concatenate(all_arrays))

Depending on the shape of your arrays and the intended concatenation, you might need to specify the axis parameter of concatenate.

Upvotes: 5

Related Questions