CODEWITHSUNDEEP
windowsmatrixmemoryout-of-memoryoctave
Yannic Bürgmann
Yannic Bürgmann

Reputation: 6581

Octave - out of memory or dimension too large for Octave's index type

I'm aware of the fact that there are 3 questions with a similar exception message. Unfortunately none of the questions is answered and the comments could not solve my problem.

I use Octave 4.2.1 in the 64 bit version on a Windows 10 System with 16 GB RAM in total and ~11 GB free during runtime. When i try to multiply a 60000 x 10 matrix with a 10 x 60000 matrix Octave comes up with the following exception:

error: out of memory or dimension too large for Octave's index type

This multiplication would result in a 60000 x 60000 matrix and therefore should not be a problem for a 64 bit index. I can't even do zeros(60000,60000);

I don't get what i am doing wrong. Could someone point me into the right direction?

Upvotes: 4

Views: 11382

Answers (1)

Tasos Papastylianou
Tasos Papastylianou

Reputation: 22225

As is often the case, this error is often misinterpreted (maybe we should address this as a bug on the octave tracker already ;) )

>> 60000*60000
ans =    3.6000e+09
>> intmax
ans = 2147483647
>> 60000*60000 > intmax
ans = 1

I.e. the number of elements of the resulting 60000x60000 matrix is larger than the maximum integer representation supported by the system, therefore there is no way to linearly index such a matrix using an integer index.

Also, in order to use actual 64-bit indexing, you need to compile octave in that manner, as this tends not to be the default, but unfortunately that's not as straightforward as you might wish, as you'll have to use the respective 64-bit supporting libraries as well. More on that here.

Having said that, it may well be possible to make use of sparse matrices instead if your matrices are indeed sparse in nature. If not, you're essentially using 'big data', and you need to find workarounds, such as blockprocessing / mapping large arrays to files etc. It's worth reading up on common "big data" techniques. Unfortunately octave does not seem to support matlab's memmapfile command just yet, but you can simulate this using fwrite / fread / fseek appropriately to read appropriate ranges from a file.

Upvotes: 5

Related Questions