CODEWITHSUNDEEP
pythonnumpymatrix
Okabe Kimtaro
Okabe Kimtaro

Reputation: 3

Initializing a big matrix with numpy taking way too long

I am working in python and I have encountered a problem: I have to initialize a huge array (21 x 2000 x 4000 matrix) so that I can copy a submatrix on it. The problem is that I want it to be really quick since it is for a real-time application, but when I run numpy.ones((21,2000,4000)), it takes about one minute to create this matrix. When I run numpy.zeros((21,2000,4000)), it is instantaneous, but as soon as I copy the submatrix, it takes one minute, while in the first case the copying part was instantaneous.

Is there a faster way to initialize a huge array?

Upvotes: 0

Views: 925

Answers (1)

wildwilhelm
wildwilhelm

Reputation: 5019

I guess there's not a faster way. The matrix you're building is quite large (8 byte float64 x 21 x 2000 x 4000 = 1.25 GB), and might be using up a large fraction of the physical memory on your system; thus, the one minute that you're waiting might be because the operating system has to page other stuff out to make room. You could check this by watching top or similar (e.g., System Monitor) while you're doing your allocation and watching memory usage and paging.

numpy.zeros seems to be instantaneous when you call it, because memory is allocated lazily by the OS. However, as soon as you try to use it, the OS actually has to fit that data somewhere. See Why the performance difference between numpy.zeros and numpy.zeros_like?

Can you restructure your code so that you only create the submatrices that you were intending to copy, without making the big matrix?

Upvotes: 1

Related Questions