Having problems with NumPy matrices

Question

I'm very familiar with MATLAB, but I'm having trouble learning Python.

The problem is creating an array X(idx,:) where I increment idx with each line I read and assign X(idx,:) to a row of data. The data, and a sample Python program for as far as I've got, follows.

File: "sample_data"

TRE-G3T- Triumph-        0.000 10/01/2013 227001.30 1760.00000 10/01/2013 227016.30   41 31 27.35998  -70 40 19.00843      -28.130        0.707   882922.244  2652775.212      -65.517      -24.677      -13.470        0.020        0.022        0.041        0.051      -13.469       -0.771   0.0109 2   1.80  7        0.005 S              0.032    -0.024     0.001  -0.256  -0.162   0.554
TRE-G3T- Triumph-        0.000 10/01/2013 227001.40 1760.00000 10/01/2013 227016.40   41 31 27.35993  -70 40 19.00854      -28.123        0.714   882922.235  2652775.207      -65.519      -24.679      -13.464        0.020        0.022        0.041        0.051      -13.463       -0.650   0.0136 2   1.80  7        0.006 S              0.005    -0.039     0.057   0.066   0.301  -0.602
TRE-G3T- Triumph-        0.000 10/01/2013 227001.50 1760.00000 10/01/2013 227016.50   41 31 27.35997  -70 40 19.00848      -28.128        0.710   882922.240  2652775.211      -65.518      -24.678      -13.468        0.020        0.022        0.041        0.051      -13.467       -0.631   0.0103 2   1.80  7        0.007 S              0.012    -0.011    -0.005   0.006  -0.094   0.606
TRE-G3T- Triumph-        0.000 10/01/2013 227001.60 1760.00000 10/01/2013 227016.60   41 31 27.36002  -70 40 19.00858      -28.122        0.715   882922.232  2652775.216      -65.520      -24.676      -13.462        0.020        0.022        0.041        0.051      -13.462       -0.829   0.0109 2   1.80  7        0.007 S              0.014    -0.021     0.056  -0.021   0.214  -0.435
TRE-G3T- Triumph-        0.000 10/01/2013 227001.70 1760.00000 10/01/2013 227016.70   41 31 27.36005  -70 40 19.00849      -28.125        0.712   882922.239  2652775.219      -65.518      -24.675      -13.465        0.020        0.022        0.041        0.051      -13.465       -1.040   0.0106 2   1.80  7        0.006 S              0.011     0.000     0.011   0.110   0.264  -0.284

Python program:

#!/usr/local/bin/python 


import numpy as np
import matplotlib.pyplot as plt


def dms2deg(deg,min,sec):
# jad - 20131103
        sgn = float(deg) / abs(float(deg))
        return sgn * (abs(float(deg)) + (float(min) + (float(sec)/60) ) / 60 )


def decdeg2dms(dd):
# http://stackoverflow.com/questions/2579535/how-to-convert-dd-to-dms-in-python
   is_positive = dd >= 0
   dd = abs(dd)
   minutes,seconds = divmod(dd*3600,60)
   degrees,minutes = divmod(minutes,60)
   degrees = degrees if is_positive else -degrees
   return (degrees,minutes,seconds)





#  indices into GrafNav output record
idx0 =  7   #  index of GPS seconds
idx1 =  8   #  beginning index for lat / lon
idx2 = 28   #  solution quality

f = open('sample_data','r')


ctr2 = -1

X=[]
X=np.array(X)



for line in f:

    var = line.split()
    l=len(var)

    if l > 35:                 #  data has more than 35 columns
        GPS_sec = var[idx0]

        lat_deg = var[idx1+0]
        lat_min = var[idx1+1]
        lat_sec = var[idx1+2]

        lon_deg = var[idx1+3]
        lon_min = var[idx1+4]
        lon_sec = var[idx1+5]

        h_ell = var[idx1+6]

        latd = dms2deg(lat_deg,lat_min,lat_sec)
        lond = dms2deg(lon_deg,lon_min,lon_sec)


        Q = var[idx2]
        stdev = var[idx2-4]
        h_sep = var[idx2-2]
        amb_drift = var[idx2-1]
        nsat = var[idx2+2]

        ctr2 += 1
             X=np.array([float(GPS_sec),float(latd),float(lond),float(h_ell),int(Q),int(nsat),float(stdev),float(h_sep),float(amb_drift)])
        print GPS_sec,  latd,  lond, h_ell,  Q, nsat, stdev, h_sep, amb_drift

f.close()

Answer 1

It appears you just want to convert your text data to numpy array. As @BiRico pointed out, you can't have more than one data type dtypein one array, if you must do that you have to use a strutured array, or better yet, pandas dataframe.

Also numpy has build IO functions genfromtxt(), which is quite fast. Use it instead of hard code one:

>>> dtypeLS=[('Var0', 'S10'),
    ('Var1', 'S10'),     
    ('Var2', 'f8'),      
    ('Var3', '<M8[D]'),  
    ('Var4', 'f8'),      
    ('Var5', 'f8'),      
    ('Var6', '<M8[D]'),  
    ('Var7', 'f8'),      
    ('Var8', 'f8'),      
    ('Var9', 'f8'),      
    ('Var10', 'f8'),     
    ('Var11', 'f8'),     
    ('Var12', 'f8'),     
    ('Var13', 'f8'),     
    ('Var14', 'f8'),     
    ('Var15', 'f8'),     
    ('Var16', 'f8'),     
    ('Var17', 'f8'),     
    ('Var18', 'f8'),     
    ('Var19', 'f8'),     
    ('Var20', 'f8'),     
    ('Var21', 'f8'),     
    ('Var22', 'f8'),     
    ('Var23', 'f8'),     
    ('Var24', 'f8'),     
    ('Var25', 'f8'),     
    ('Var26', 'f8'),     
    ('Var27', 'f8'),     
    ('Var28', 'f8'),     
    ('Var29', 'f8'),     
    ('Var30', 'f8'),     
    ('Var31', 'f8'),     
    ('Var32', 'S10'),    
    ('Var33', 'f8'),     
    ('Var34', 'f8'),     
    ('Var35', 'f8'),     
    ('Var36', 'f8'),     
    ('Var37', 'f8'),     
    ('Var38', 'f8')]
>>> a=genfromtxt('temp.txt', dtype=dtypeLS)
>>> a[0]
('TRE-G3T-', 'Triumph-', 0.0, datetime.date(2013, 10, 1), 227001.3, 1760.0, datetime.date(2013, 10, 1), 227016.3, 41.0, 31.0, 27.35998, -70.0, 40.0, 19.00843, -28.13, 0.707, 882922.244, 2652775.212, -65.517, -24.677, -13.47, 0.02, 0.022, 0.041, 0.051, -13.469, -0.771, 0.0109, 2.0, 1.8, 7.0, 0.005, 'S', 0.032, -0.024, 0.001, -0.256, -0.162, 0.554)
>>> a['Var11']
array([-70., -70., -70., -70., -70.])
>>> a['Var12']
array([ 40.,  40.,  40.,  40.,  40.])
>>> a['Var13']
array([ 19.00843,  19.00854,  19.00848,  19.00858,  19.00849])
>>> np.sign(a['Var11'])*(np.abs(a['Var11'])+a['Var12']/60+a['Var13']/3600)
array([-70.67194679, -70.67194682, -70.6719468 , -70.67194683, -70.6719468 ])     

Finally, just like in Matlab, always vectorize if you want things to run fast. See the dd-to-dms converting code in the last line.

Also, I have to change your day format from 10-01-2013 to 2013-10-01 to use datetime dtype.

Answer 2

You'll get much better answers if you can focus your question on what you're really asking. Your code has lot of stuff that's irrelevant to the question that's hard to dig though to get to the real meat of the stuff. I think you want something like this:

X = []
for line in f:
    ....
    row = np.array([GPS_sec, latd, lond, h_ell, Q, nsat, stdev, h_sep, amb_drift],
                   dtype=float)
    X.append(row)

X = np.array(X)

What this does is hold the rows of your final array in a list and then convert it to an array after the loop. Also you should know that ndarrays can only have one dtype, I believe that's the same as matlab, so it's not clear what you're trying to accomplish by doing np.array([float(a), int(b)]). Hope this helps.