CODEWITHSUNDEEP
pythonbashjupyter-notebook
Darien Marks
Darien Marks

Reputation: 535

Bash command won't execute in Jupyter Notebook

I am trying to execute a Bash command from inside of a Jupyter Notebook that writes a random number to some files. The command I'd like to issue is 

echo $(( RANDOM )) &> {output_files[i]}

where output_files is a Python list of output filenames. I can't figure out how to put the Python variable output_files[i] into a Bash command successfully, so instead I create the entire command as a string and try to execute that through the Notebook:

# Create a list of 5 filenames
output_files = [os.path.join(os.getcwd(), 'random-%s.txt' % i) for i in range (5)]

# Write random number to each file
for i in range (5):
    command = "echo $(( RANDOM )) &> " + output_files[i]
    print(command)
    !{command}

When I try this, the files ./random-0.txt etc. are created, but no number is written to them - they are empty. When I run the created commands directly in the terminal, though, it works exactly as expected. The files are created and contain a single random number each. The print(command) line produces

echo $(( RANDOM )) &> /filepath/random-0.txt
0
echo $(( RANDOM )) &> /filepath/random-1.txt
echo $(( RANDOM )) &> /filepath/random-2.txt
echo $(( RANDOM )) &> /filepath/random-3.txt
echo $(( RANDOM )) &> /filepath/random-4.txt

Web searches for this problem give hits for several similar questions on Stack Overflow as well as blogs, but the result of all of them is either "Bash commands don't exist on Windows" or "You can use '!' to execute Bash commands in Jupyter Notebooks!" with no further information about how it works so that I could debug an issue like this.

How do I make this work?

Upvotes: 3

Views: 2719

Answers (3)

tripleee
tripleee

Reputation: 189387

In reaction to one of the answers here, let me show how to do this with subprocess.check_call().

If you want to use a shell redirection in subprocess, you have to have a shell:

for i in range(5):
    subprocess.check_call('echo "$RANDOM" >random-{}.txt'.format(i), shell=True)

You could superficially try to get rid of the shell=True ... but then echo and $RANDOM are shell features, too.

A pure-Bash solution would be

!for i in {0..4}; do printf "%s\n" "$RANDOM" >random-"$i".txt; done

(As an aside, the numeric context in $((RANDOM)) is not wrong, but entirely superfluous.)

You can interpolate a Python expression into the ! command line with {...}. The following also works, for example:

!{';'.join(['echo $RANDOM >random-{}.txt'.format(i) for i in range(5)])}

In iPython I can produce the results I expect with the construct you tried:

in [15]: for i in range(2):
    ...:     c='echo $RANDOM >random-{}.txt'.format(i)
    ...:     !{c}
    ...:

in [16]: !tail random-*.txt
==> random-0.txt <==
15637

==> random-1.txt <==
32475

If Jupyter is using the iPython kernel, I'm thinking this ought to work for you, too.

Notice also how I used plain old > redirection rather than &> - there really really really is no sane reason to want error messages to be redirected to the files; and if a file can't be created, you have absolutely no idea what failed or why, or even that it failed. (I suspect this is really the reason for your problems, actually. Permission denied? Disk full? We can't know.)

I would suggest you use Python to perform this simple task, though. Running a subprocess to perform something which is relatively easy in Python itself is inefficient and clumsy as well as somewhat brittle.

Upvotes: 2

Inian
Inian

Reputation: 85580

Why not do the whole thing in python itself? Use the random module the randint() function to generate your numbers. Define a range of your choice and include it within the (..)

import os
import random
output_files = [os.path.join(os.getcwd(), 'random-%s.txt' % i) for i in range (5)]
for of in output_files:
    with open(of, "w") as text_file:
        text_file.write(str(random.randint(99,1000)))

Upvotes: 3

Heiko Becker
Heiko Becker

Reputation: 576

Hello and welcome to Stack Overflow,

As far as I can tell, your issue does not arise form the bash command execution with ! but rather from the fact that you do not "replace" output_files[i] by its actual content. When you run the loop

# Write random number to each file
for i in range (5):
    command = "echo $(( RANDOM )) &> {output_files[i]}"

in each iteration, command will be the literal string "echo $(( RANDOM )) &> {output_files[i]}". As you did in your first line, you should use string substitution to replace a placeholder with the content of output_files[i]:

# Write random number to each file
for i in range (5):
    command = ("echo $(( RANDOM )) &> {%s}"%output_files[i])

To debug such an issue, it is usually a good start to inspect the value of the variable in question. The easiest way of doing so is by adding print (x) lines where x is the variable whose content you want to inspect.

Upvotes: 2

Related Questions