how to extract elements from a list in python given filter criterion

Question

I have this list:

data = [[0.0322249406353, 1.00005691884],
[0.0322267803497, 0.999999986608],
[0.0322286200641, 0.499997756763],
[0.0322304597785, 0.333330346815],
[0.0322322994929, 0.249996641841],
[0.0322341392073, 0.199996418857],
[0.0322359789217, 0.166662936867],
[0.0322378186361, 0.142853306874],
[0.0322396583505, 0.12499608438],
[0.032241498065, 0.111107133551],
[0.0322433377794, 0.0999959728877],
[0.0322451774938, 0.0909050232541],
[0.0322470172082, 0.0833292318929],
[0.0322488569226, 0.0769189468948],
[0.032250696637, 0.0714244168966],
[0.032350696637, 0.],
[0.032450696637, -0.04]]

Because I'll use it later and I don't want to use a list with many information clustered in a particular range and scarce information in the rest of the data I'd like to filter my original list so that I end up with a list where the distance bewteen consecutive values of data[i][1] (second column) is larger than a given value, say 0.05, if they are within [0,1]. data is a list where the second column values continuously decrease, so data[i][1] < data[i-1][1]. So the list that I want is something like:

data2 = [[0.0322249406353, 1.00005691884],
[0.0322267803497, 0.999999986608],
[0.0322286200641, 0.499997756763],
[0.0322304597785, 0.333330346815],
[0.0322322994929, 0.249996641841],
[0.0322378186361, 0.142853306874],
[0.032350696637, 0.],
[0.032450696637, -0.04]]

Any ideas how this can be done? Thanks

EDIT (first attempt):

data2=[] 
for i in xrange(0,len(data)): 
    if 0>data[i][1] or data[i][1]>1:
        data2.append([data[i][0],data[i][1]])
    for j in xrange(0,len(data)):
        if j>i and 0<data[i][1]<1 and 0<data[j][1]<1:
            if data[i][1] - data[j][1] > 0.05:
                data2.append([data[i][0],data[i][1]])
                i = j
                break

This works partially, because I get an incomplete and incorrect new list:

data2=[[0.0322267803497, 0.999999986608], [0.0322286200641, 0.499997756763], [0.0322304597785, 0.333330346815], [0.0322322994929, 0.249996641841], [0.0322341392073, 0.199996418857], [0.0322359789217, 0.166662936867], [0.0322378186361, 0.142853306874], [0.0322396583505, 0.12499608438], [0.032450696637, -0.04]]

Answer 1

You should make generators for filter your list. Below, an example of usage for get only lists which have a pair number as first:

l = [ [0,1], [1,2], [2,3], [3,4] ]
def get_first_divisible_by_2(l):
    for i,j in l:
        if not i % 2:
            yield i,j
[ c for c in get_first_divisible_by_2(l) ]
[(0, 1), (2, 4)]

or a generator comprehension:

list( ( (i,j) for i,j in l if not i % 2 ) )
[(0, 1), (2, 4)]

Simply adapt it with your filter and maybe with coroutines, you'll be able to handle easily previous result.

Answer 2

Here are a few tips:

• Apparently, you want to go through a data structure one by one, checking a predicate for every element, so what kind of loop could help you with this?
• For each element you want to compare the next one to the previous one and check whether the difference between two elements is greater than a specific value, so how could your predicate look like?
• And last but not least, what do you want to do with an item if your predicate returns true?

EDIT:

Using a for loop is the correct strategy, albeit nesting is not really necessary here. You can simply add any value to the the new list if it is greater 1 or smaller 0. For every other element you simply have to check against the newly generated list if the difference between the current last element in the result list and the current element to check is within or without of the boundary and then add it or ignore it.

EDIT 2:

Here is a possible solution:

data2=[]
limit=0.1
j = 0
# get all values > 1 in the result list
while data[j][1] > 1:
    data2.append(data[j])
    j = j + 1
# the next one too
data2.append(data[j])
for i in xrange(0,len(data)): 
# compare current to last in results and see if it is smaller than the limit
    if abs(data[i][1] - data2[len(data2)-1][1]) > limit and data[i][1] > 0:
        data2.append(data[i])
        j = i
# in the end add all the elements < 0
for i in xrange(0,len(data)):
    if data[i][1] < 0:
        data2.append(data[i])