CODEWITHSUNDEEP
pythonarraysnumpy
Richard Rublev
Richard Rublev

Reputation: 8172

How to convert itertools.chain into numpy array?

My code

import math
import itertools

with open('/home/milenko/OCCAM2DMT_V3.0/13042/ITER04.iter') as f:
    lines_after_19 = f.readlines()[19:]
    p = []
    for line in lines_after_19:
        line = line.split()         
        line = [float(i) for i in line]
        p.extend(line)    

a1=p[1:81]
for i in a1:
    b1=math.pow(10, i)   
a2=p[83:163]
for i in a2:
    b2=math.pow(10,i)
a3=p[165:245]   
for i in a3:
    b3=math.pow(10,i)
a4=p[247:327]
for i in a4:
    b4=math.pow(10,i)
a5=p[329:409] 
for i in a5:
    b5=math.pow(10,i)
a6=p[411:491]
for i in a6:
    b6=math.pow(10,i)

c = itertools.chain(b1, b2, b3, b4, b5, b6) 
print type(c)

I need to convert c to numpy array,for reshape and then further calculations.

Upvotes: 2

Views: 2038

Answers (1)

hpaulj
hpaulj

Reputation: 231385

This was marked as duplicate with a reference to this: How do I build a numpy array from a generator? question.

The highest ranking answer, using fromiter is a good choice when dealing with a generator, but that doesn't really address the issue in this question.

As commented, the for loops don't save anything except the last calculation. And they needlessly perform math.pow calculations that could just as well be done in numpy.

Skipping the pow bit for now:

In [36]: ll=range(100)
In [53]: a=ll[0:20]    
In [54]: b=ll[22:40]
In [55]: c=ll[42:60]

There are several ways I could combine these into a one array; first just a list.

In [56]: len(list(itertools.chain(a,b,c)))
Out[56]: 56

an array from that list

In [57]: np.array(list(itertools.chain(a,b,c)))
Out[57]: 
array([ 0,  1,  2,  3,  4,  5,...., 59])

fromiter is a bit faster;

In [58]: np.fromiter(itertools.chain(a,b,c),int)
Out[58]: 
array([ 0,  1,  2,  3,  4,  5, ... 58, 59])

so is concatenate - it converts its inputs to arrays and concatenates them (default is flat):

In [59]: np.concatenate((a,b,c))
Out[59]: 
array([ 0,  1,  2,  3,  4,  5, ... 52, 53, 54,
       55, 56, 57, 58, 59])

It's easy to perform the pow calculation on the array

In [62]: 10**(la/100.)
Out[62]: 
array([ 1.        ,  1.02329299,  1.04712855,  1.07151931,  1.0964782 ,
        1.12201845,  1.14815362,  1.17489755,  1.20226443,  1.23026877,
        ...
        3.4673685 ,  3.54813389,  3.63078055,  3.71535229,  3.80189396,
        3.89045145])

Since the numbers have been read in from a file, and are present in the list p, there isn't much point in trying to save memory by using a generator. chain is useful here just as a way of creating a flat list.

If all lines in the file had the same number of items, it could have been loaded with a np.loadtxt. The result would be a 2d array.

All your sublists are length 80. Are there 81 numbers in each line, and you are skipping the first? If so, the loadtxt array would be (N,81) in shape, and you could easily drop the 1st with slicing [:, 1:].

Since all the lists that you chain are the same length you could have just passed those to np.array without chain.

 np.array((b1, b2, b3, b4, b5, b6))

would produce a (6,80) shaped array. No need to reshape - unless you want a different one.

Upvotes: 7

Related Questions