CODEWITHSUNDEEP
pythonarraysnumpymergeconcatenation
Jasper Vaneessen
Jasper Vaneessen

Reputation: 43

Effective way to reduce dimension of an n-dimensional numpy array

I'm working on a project where I have a set of samples for gesture recognition (ChaLearn dataset).

The input is a (10000,6,32,64,64) array. I have 10000 examples, 6 featuremaps (body, l-hand, r-hand in depth and rgb images), 32 frames of 64 by 64.

I have to select only the depth images (which are indices 1, 3 and 5) and 4 frames out of 32 (indices 8,13,18,25)

I'm not that familiar with python so I was messing around and I think my solution is fine but it is really slow. Mostly the two selections take up alot of time (70-80 seconds for this bit).

samples.shape gives me (10000,6,32,64,64) at the start

samples = samples[0:samples.shape[0],0:samples.shape[1],(7,13,18,25)]
samples = samples[0:samples.shape[0],(1,3,5)]
samples = floatX(samples.reshape((samples.shape[0],12,64,64)))/255.

samples.shape gives me (10000,12,64,64) at the end.

Is there a more efficient way of accomplishing my goal? Or am I completely wrong here?

ignore the floatx part for now :)

Upvotes: 2

Views: 3023

Answers (3)

Hooked
Hooked

Reputation: 88198

Is there a more efficient way of accomplishing my goal?

If you are willing to look into another layer of abstraction, you might want to consider a format that is designed for larger data. Personally, I use HDF which has a few nice python modules like h5py and pytables.

Two main advantages I see, you'll be able to store much larger datasets at once (since you don't have to load the entire set into memory) and you'll be able to apply meta-data to the data (with .attrs). When you build the hdf data set you can organize it so your "query" of something like "indices 1, 3 and 5" can be much more logical.

Upvotes: 1

Jaime
Jaime

Reputation: 67447

As mentioned in the other answer, there are some discussions in numpy about making this easier, but in your case, since the two axes you want to index along are adjacent in the shape, things aren't so bad. This should get you what you are after avoiding that intermediate copy altogether:

depth_idx = np.array([1, 3, 5], dtype=np.intp)
frame_idx = np.array([8, 13, 18, 25], dtype=np.intp)
samples = samples[:, depth_idx[:, None], frame_idx].reshape(-1, 12, 64, 64)

Actually, since your depth_idx happens to be writeable as a slice, for this particular case you will actually be better off doing:

samples = samples[:, 1::2, frame_idx].reshape(-1, 12, 64, 64)

Upvotes: 2

TheBlackCat
TheBlackCat

Reputation: 10308

First, you can just do samples[:,:,(7,13,18,25)]. However, the reason this is slow is because you are doing you are doing 2 copy operations. You can't currently get around this, although you can simplify stuff somewhat using take:

samples = np.take(np.take(samples, [7,13,18,25], axis=2), [1,3,4], axis=1)

There is actually work going on to make the operation you are trying to do much easier, but it isn't ready yet.

If you aren't using numpy 1.9, you should definitely upgrade to it, since the "fancy" indexing you are doing is much faster. The take version will be faster in earlier versions.

Upvotes: 1

Related Questions