CODEWITHSUNDEEP
pythonpython-2.7python-3.x
Vedant
Vedant

Reputation: 523

Splitting a large file in python by calling a command line function

I am trying to split a file into a number of parts via a python script:

Here is my snippet:

def bashCommandFunc(commandToRun):
    process     = subprocess.Popen(commandToRun.split(), stdout=subprocess.PIPE)    
    output      = process.communicate()  
    return output


filepath = "/Users/user/Desktop/TempDel/part-00000"
numParts = "5"

splitCommand = "split -l$((`wc -l < " + filepath + "/" + numParts + ")) " + filepath 


splitCommand:
'split -l$((`wc -l < /Users/user/Desktop/TempDel/part-00000`/5)) /Users/user/Desktop/TempDel/part-00000'

If I run this command on a terminal, it splits the file as it's supposed to, but it fails for the above defined subprocess function.

I have tested the function for other generic commands and it works fine.

I believe the character " ` " (tilde) might be an issue,

What is the work around to getting this command to work? Are there some better ways to split a file from python into "n" parts.

Thanks

Upvotes: 0

Views: 357

Answers (1)

viraptor
viraptor

Reputation: 34145

You'll have to let Python run this line via a full shell, rather than trying to run it as a command. You can do that by adding shell=True option and not splitting your command. But you really shouldn't do that if any part of the command may be influenced by users (huge security risk).

You could do this in a safer way by first calling wc, getting the result and then calling split. Or even implement the whole thing in pure Python instead of calling out to other commands.

What happens now is that you're calling split with first parameter -l$((``wc, second parameter -l, etc.

Upvotes: 1

Related Questions