CODEWITHSUNDEEP
bashshellstdin
user1265125
user1265125

Reputation: 2656

Mocking stdin as a file

I have a Python script which needs a --file xyz.json argument.

Thing is, my JSON is immense, hence it is Gzipped. Of course I could un-gzip it and then run the script, but that seems wasteful. Is there a clever way I can get this to work while doing a zcat xyz.json.gz | myscript.py --file ?????. I don't want to go into modifying myscript.py to read stdin instead of a file unless there's no way to get this done otherwise.

Thanks!

Upvotes: 0

Views: 219

Answers (2)

joepd
joepd

Reputation: 4841

Try:

myscript.py --file <(zcat xyz.json.gz)

A file descriptor containing the pipe is returned. Provided that the script just reads the file, and does not search forward and backward, this should work.

The <( ... ) is called process substitution.

As an elaboration on what happens:

% awk 'BEGIN{print "filename:", ARGV[1]};1'  <(echo first; sleep 1; echo second) 
filename: /proc/self/fd/11
first
second

The second gets printed after a delay. So: Awk gets the filename /proc/self/fd/11, and starts to process it. It will immediately see the first line, and print it out. Then, after the sleep, it will see the second line, and print it as well.

Upvotes: 4

ruakh
ruakh

Reputation: 183301

You can use /dev/stdin or (equivalently) /dev/fd/0:

zcat xyz.json.gz | myscript.py --file /dev/stdin
zcat xyz.json.gz | myscript.py --file /dev/fd/0

Upvotes: 2

Related Questions