split with specific comma not within quotes in python

Question

i have this string and i want to split it with ","

x = 'a, b, c , d , "x,x,2" , hi'
x.split(',')

here is my real string

x = 'Outward   ,Supply , ,Tax Invoice ,IN9195212470,31/12/2019,VPS AGRO & AUTO PVT LTD ,311954,06AAACV9344F1ZA ,"VILLAGE KHANPUR KOLIAN, N.H. 1 ",6 K.M. FRO,KURUKSHETRA   ,HARYANA ,136131,VPS AGRO & AUTO PVT LTD ,311954,"VILLAGE KHANPUR KOLIAN, N.H. 1",6 K.M. FRO,KURUKSHETRA                             ,HARYANA             ,136131,503675,SM VAL. GENUINE DIESEL ENG. OIL 1/9 L   ,27101980,360,LTR,58204.04,9,5238.36,9,5238.36,0,0,0,0,0,0,0,68680.76,                    ,                    ,                              ,          ,          ,,               ,          ,06AAACW0287A1ZR   ,VALVOLINE CUMMINS PVT LTD-AMBALA        ,"KHASHRA NO-108/1/2,                                         ",          ,AMBALA                                  ,133004,HARYANA             ,                    ,                  ,                              , ,'

it returns this result

['a','b','c','d','"x','x','2', 'hi']

But I want to have this one

['a', 'b', 'c' , 'd' , '"x,x,2"' , 'hi']

how can do this in python

Help me

Answer 1

import shlex
lexer = shlex.shlex('a, b, c , d , "x,x,2" , hi')
lexer.whitespace += ','
print(list(lexer))

Result:

['a', 'b', 'c', 'd', '"x,x,2"', 'hi']

Here’s an updated solution for updated task:

x = 'Outward   ,Supply , ,Tax Invoice ,IN9195212470,31/12/2019,VPS AGRO & AUTO PVT LTD ,311954,06AAACV9344F1ZA ,"VILLAGE KHANPUR KOLIAN, N.H. 1 ",6 K.M. FRO,KURUKSHETRA   ,HARYANA ,136131,VPS AGRO & AUTO PVT LTD ,311954,"VILLAGE KHANPUR KOLIAN, N.H. 1",6 K.M. FRO,KURUKSHETRA                             ,HARYANA             ,136131,503675,SM VAL. GENUINE DIESEL ENG. OIL 1/9 L   ,27101980,360,LTR,58204.04,9,5238.36,9,5238.36,0,0,0,0,0,0,0,68680.76,                    ,                    ,                              ,          ,          ,,               ,          ,06AAACW0287A1ZR   ,VALVOLINE CUMMINS PVT LTD-AMBALA        ,"KHASHRA NO-108/1/2,                                         ",          ,AMBALA                                  ,133004,HARYANA             ,                    ,                  ,                              , ,'


import shlex
lexer = shlex.shlex(x)
lexer.whitespace = ','
lexer.whitespace_split = True
print([cell.strip() for cell in lexer])

Result:

['Outward', 'Supply', '', 'Tax Invoice', 'IN9195212470', '31/12/2019', 'VPS AGRO & AUTO PVT LTD', '311954', '06AAACV9344F1ZA', '"VILLAGE KHANPUR KOLIAN, N.H. 1 "', '6 K.M. FRO', 'KURUKSHETRA', 'HARYANA', '136131', 'VPS AGRO & AUTO PVT LTD', '311954', '"VILLAGE KHANPUR KOLIAN, N.H. 1"', '6 K.M. FRO', 'KURUKSHETRA', 'HARYANA', '136131', '503675', 'SM VAL. GENUINE DIESEL ENG. OIL 1/9 L', '27101980', '360', 'LTR', '58204.04', '9', '5238.36', '9', '5238.36', '0', '0', '0', '0', '0', '0', '0', '68680.76', '', '', '', '', '', '', '', '06AAACW0287A1ZR', 'VALVOLINE CUMMINS PVT LTD-AMBALA', '"KHASHRA NO-108/1/2,                                         "', '', 'AMBALA', '133004', 'HARYANA', '', '', '', '']

Answer 2

There are no built-ins that can achieve that without a lot of hacking to pre/post process the data.

• shlex.split would work to some extent with this example, but it's cheating, since it splits on spaces. If 2 elements are collated with only commas, this will fail.
• ast.literal_eval won't work since ... some items aren't literals
• csv.reader object almost does the trick with [x.strip() for x in next(csv.reader([x]))] but quotes aren't processed properly because there are spaces between them and commas.

But looping through each character with a simple state machine this can be done:

x = 'a, b, c , d , "x,x,2" , hi'

in_quote = False
current = []
output = []
for c in x:
    if in_quote:
        current.append(c)
        if c=='"':
            output.append("".join(current))
            current = []
            in_quote = False
        continue

    if c==",":
        output.append("".join(current))
        current = []
    elif c==" ":
        pass
    else:
        current.append(c)
        if c=='"':
            in_quote = True

output.append("".join(current))

result:

['a', 'b', 'c', 'd', '"x,x,2"', '', 'hi']

Just skip spaces, create a new element when comma is encountered, but have a flag if quote is encountered.

In the end, don't forget the last element being accumulated when end of string is encountered.

Answer 3

You could use a regex approach:

import regex as re

x = 'a, b, c , d , "x,x,2" , hi'

rx = re.compile(
    r"""
    "[^"]*"(*SKIP)(*FAIL)
    |
    \s*,\s*
    """, re.VERBOSE)
lst = rx.split(x)
print(lst)

This yields

['a', 'b', 'c', 'd', '"x,x,2"', 'hi']

Answer 4

A solution using only split. Please note it uses f strings (python 3.6+) but the same behaviour can still be achieved in older versions. It is possible to achieve this without using regex as follows: I will comment the code for explanation:

# First split by double quote
x = x.split('"')
final_x = []
for i in range(len(x)):
    # We know that if the list element is even then it must be outside double quotes
    if i%2 == 0:
        # Split the list by commas and strip any whitespace
        x_element = x[i].split(',')
        x_element = [el.strip() for el in x_element]
        # extend the list
        final_x.extend(x_element)
    else:
        # This is an odd element of the list, therefore inside quotation.
        # put the string back into quotations
        x_element = f'"{x[i]}"'
        #append this to the final list
        final_x.append(x_element)
# filter out any white spaces left from the various splits         
final_x = [el for el in final_x if el !=''] 

Note the difference in appending the odd list elements and extending the even. This is because you are creating a new list with the split and we want to extend our output whilst for odd elements we want to add a new element to the list, hence we append.