CODEWITHSUNDEEP
pythonnumpylinear-algebra
Steve
Steve

Reputation: 1

Can I trim data from a matrix with a partially full last row in Python?

I am currently working with a vector of variable length (on the order of 500,000 elements) and I am trying to reshape it using numpy. My code is below. 

tm = 1800 #Number of samples per row (total number of columns)
xyzd = np.genfromtxt(filename, delimiter=',', usecols=(4,5,6,14), encoding='ISO-8859–1', skip_header=1) #reading in the data
xtm = np.reshape(xyzd[:,0],(tm,len(xyzd)//tm+1)) #my attempt to resize

I get this as an error:

ValueError: cannot reshape array of size 488200 into shape (1800,272) I believe this is because the reshaping leaves the last row only partially full.

Is there a way that I can either tell reshape to only fill until it fills it's last full row and then just toss the rest of the data away or to reshape in some way that allows for it to accept that reshaped matrix, which I can then trim the last row off of.

Upvotes: 0

Views: 192

Answers (1)

Steve
Steve

Reputation: 1

It appears, as hpaulj suggested in the comments, that reshape is too dumb to have an easy way to do this. So my solution to this just involved doing some math.

columns = len(xyzd)//tm
elements = tm * columns
initial_length = len(xyzd)
n = abs(initial_length-elements)
xyzd = xyzd[:-n, :]

This solved my problem. I am making an assumption that the last few hundred data points of my dataset aren't really that important, but we are sampling at such a high rate, that we're trimming something like 2 minutes off a 5-day sample, but I'm willing to risk that nothing cool will happen then.

Upvotes: 0

Related Questions