CODEWITHSUNDEEP
pythonpython-3.x
user3884301
user3884301

Reputation:

Tokenize a list

I like to tokenize a list using a list item as a delimiter.

Is there is a pythonic way to do this or do I have to write something on my own.

Data=['Label',23,'NORM','|','RESP',1.256,None,'|','','','|','RELV','','']
SubList = TokenizeList (Data,Delim='|')

printing SubList would result in

[ ['Label',23,'NORM'] , ['RESP',1.256,None] , ['',''] , ['RELV','',''] ]

Upvotes: 3

Views: 207

Answers (3)

shaik moeed
shaik moeed

Reputation: 5785

Try this, which is simple and straight (Pythonic as well),

def tokenize_list(array, sep='|'):
    result = []
    _temp = []
    for el in array:
        if el == sep:
            result.append(_temp)
            _temp = []
        else:
            _temp.append(el)
    if _temp: # Finally append list after for-loop, to store last vlaues present in _temp if exists.
        result.append(_temp) 
    return result

Output:

>>> data = ['Label',23,'NORM','|','RESP',1.256,None,'|','','','|','RELV','','', '|']
>>> tokenize_list(data)
[['Label', 23, 'NORM'], ['RESP', 1.256, None], ['', ''], ['RELV', '', '']]

Upvotes: 1

norok2
norok2

Reputation: 26886

Try this:

def group_by_sep(items, sep='|'):
    inner_list = []
    for item in items:
        if item == sep:
            yield inner_list
            inner_list = []
        else:
            inner_list.append(item)
    if inner_list:
        yield inner_list


Data=['Label',23,'NORM','|','RESP',1.256,None,'|','','','|','RELV','','','|','|','now','|']

SubList = list(group_by_sep(Data, '|'))
print(SubList)
# [['Label', 23, 'NORM'], ['RESP', 1.256, None], ['', ''], ['RELV', '', ''], [], ['now']]

Note that a itertools.groupby approach can be used here, but it is not equivalent to the above and offers less control over the exact behavior:

import itertools


def group_by_sep2(items, sep='|'):
    yield from (
        list(g)
        for k, g in itertools.groupby(items, key=lambda x: x == sep)
        if not k)


SubList2 = list(group_by_sep2(Data, '|'))
print(SubList2)
# [['Label', 23, 'NORM'], ['RESP', 1.256, None], ['', ''], ['RELV', '', ''], ['now']]

It is missing the empty list between two consecutive separators.

Additionally, it is not as efficient as the direct method from above:

%timeit list(group_by_sep(Data))
# 1000 loops, best of 3: 1.47 µs per loop
%timeit list(group_by_sep2(Data))
# 100 loops, best of 3: 4.01 µs per loop

%timeit list(group_by_sep(Data * 1000))
# 1000 loops, best of 3: 1.33 ms per loop
%timeit list(group_by_sep2(Data * 1000))
# 100 loops, best of 3: 2.83 ms per loop

%timeit list(group_by_sep(Data * 1000000))
# 1000 loops, best of 3: 1.67 s per loop
%timeit list(group_by_sep2(Data * 1000000))
# 100 loops, best of 3: 3.22 s per loop

And the benchmarks indicate that the direct approach is ~2x to ~3x faster.

(EDITED to write it all as generators and included more edge cases)

Upvotes: 2

Sayandip Dutta
Sayandip Dutta

Reputation: 15872

Yes, you can use itertools.groupby:

>>> from itertools import groupby
>>> Data=['Label',23,'NORM','|','RESP',1.256,None,'|','','','|','RELV','','']
>>> [list(g) for k,g in groupby(Data,key=lambda x:x == '|') if not k]
[['Label', 23, 'NORM'], ['RESP', 1.256, None], ['', ''], ['RELV', '', '']]

You can make a function of course:

def splitList(sequence, delimiter):
    return [list(g) for k, g in groupby(sequence, key = lambda x: x == delimiter) if not k]
>>> splitList(sequence = Data, delimiter = '|')
[['Label', 23, 'NORM'], ['RESP', 1.256, None], ['', ''], ['RELV', '', '']]

Upvotes: 3

Related Questions