CODEWITHSUNDEEP
c++image-processing
a3dsfcv
a3dsfcv

Reputation: 1274

How to store a big matrix in C++

I'm writing a program about image-processing. I need to store an int square matrix with the size of 480 000 columns and 480 000 rows. Any ideas how can I do that?

Upvotes: 2

Views: 4395

Answers (5)

Emile Cormier
Emile Cormier

Reputation: 29209

If you need to work on the whole matrix at the same time, and most of the matrix elements are going to be blank, then you should consider using some kind of sparse matrix data structure. Many linear algebra libraries support sparse matrices (Boost.uBlas, Eigen, etc), as well as some image processing libraries (OpenCV, etc).

Upvotes: 3

bartgol
bartgol

Reputation: 1873

If it's a sparse matrix and you need to do some linear algebra on that, I would use some scientific linear algebra library like Trilinos (using Epetra or Tpetra packages) or Hypre. These are highly parallel library (which is nice if you can run your code in parallel). I have never used Hypre (though I've heard is somehow better performing than Trilinos), so I can't tell you anything about it. Trilinos is a HUGE (I would say too huge) library, with about 50-60 packages and it's not super easy to learn; but if you have to deal with huge matrices, it makes sense to rely on some TPL which is well tested and developed. For just matrices storage, Epetra/Tpetra are the packages to look into in Trilinos.

Upvotes: 0

Walter
Walter

Reputation: 45414

Don't use a 480,000 x 480,000 matrix.

The only reason to ever have this full matrix (assuming it is not sparse) is to have random access (i.e. be able to access any element at any time). Even if you can somehow achieve this (storing 0.9Tb), the data access will be extremely slow (in particular when mapping it to file), making your algorithm inefficient.

Instead, think of a way to re-write your algorithm such that it doesn't need random access to the whole matrix at any time, but perhaps only to a small part of it, which you create (and then delete) when needed, or any other way of reducing the need to store this many data.

High performance is not just about a reduction of the amount of computing, but crucially also about the reduction in random data access.

Upvotes: 4

fbafelipe
fbafelipe

Reputation: 4952

It depends on the characteristics the matrix will have.

Will it have a lot of 0? If so, you can use a sparse matrix implementation, which do not store 0s.

If it's a band matrix you can store just the diagonal band.

You will have to look to the matrix properties and see where you can save memory. If you can't find any property that allow such optimizations, then you will have to store it on a file.

Upvotes: 1

Some programmer dude
Some programmer dude

Reputation: 409166

You can store it in a file, and map the portions of the matrix you need into memory. See e.g. http://en.wikipedia.org/wiki/Memory-mapped_file

Upvotes: 3

Related Questions