CODEWITHSUNDEEP
pythonlinuxbashshellsubprocess
user2467545
user2467545

Reputation:

How to make a big shell script in a single line shell script in json document?

I am working on Python along with bash shell script. I need to execute shell script from the Python script. I am successfully able to do that.. I need to have a shell script in a single line in a JSON String..

Below is an example which is working fine for a simple shell script which I have made it in a JSON document...

import os
import json
import subprocess

jsonData = '{"pp": [0,3,5,7,9], "sp": [1,2,4,6,8]}'
jj = json.loads(jsonData)

os.putenv( 'jj3', ' '.join( str(v) for v in jj['pp']  ) )
os.putenv( 'jj4', ' '.join( str(v) for v in jj['sp']  ) )

jsonStr = '{"script":"#!/bin/bash \\n echo Hello World \\n "}'

j = json.loads(jsonStr)

shell_script = j['script']

print "start"
proc = subprocess.Popen(shell_script, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdout, stderr) = proc.communicate()
if proc.returncode != 0:
    print "Shell script gave some error"
    print stdout
else:
    print "end"
    print stdout

Now I have a below shell script which I need to represent it in a single line in a JSON document as I have done for above Hello World use case..

#!/bin/bash
set -e

readonly PRIMARY=/tech01/primary
readonly SECONDARY=/tech02/secondary
readonly LOCATION=(machineA machineB)
readonly MAPPED_LOCATION=/bat/data/snapshot
HOSTNAME=$hostname

dir1=$(ssh -o "StrictHostKeyChecking no" david@${LOCATION[0]} ls -dt1 "$MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${LOCATION[1]} ls -dt1 "$MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1
echo $dir2

length1=$(ssh -o "StrictHostKeyChecking no" david@${LOCATION[0]} "ls '$dir1' | wc -l")
length2=$(ssh -o "StrictHostKeyChecking no" david@${LOCATION[1]} "ls '$dir2' | wc -l")

echo $length1
echo $length2

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
    rm -rf $PRIMARY/*
    rm -rf $SECONDARY/*
    for el in $primary_partition
    do
        scp david@${LOCATION[0]}:$dir1/weekly_8880_"$el"_5.data $PRIMARY/. || scp david@${LOCATION[1]}:$dir2/weekly_8880_"$el"_5.data $PRIMARY/.
    done
fi

I have made the above shell script in a single line JSON document like this but this doesn't work and I always get an error

jsonStr = '{"script":"#!/bin/bash \n set -e \n readonly PRIMARY=/tech01/primary \n readonly SECONDARY=/tech02/secondary \n readonly LOCATION=(machineA machineB) \n readonly MAPPED_LOCATION=/bat/data/snapshot \n HOSTNAME=$hostname \n dir1=$(ssh -o \"StrictHostKeyChecking no\" david@${LOCATION[0]} ls -dt1 \"$MAPPED_LOCATION\"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1) \n dir2=$(ssh -o \"StrictHostKeyChecking no\" david@${LOCATION[1]} ls -dt1 \"$MAPPED_LOCATION\"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1) \n echo $dir1 \n echo $dir2 \n length1=$(ssh -o \"StrictHostKeyChecking no\" david@${LOCATION[0]} \"ls \'$dir1\' | wc -l\") \n length2=$(ssh -o \"StrictHostKeyChecking no\" david@${LOCATION[1]} \"ls \'$dir2\' | wc -l\") \n echo $length1 \n echo $length2 \n if [ \"$dir1\" = \"$dir2\" ] && [ \"$length1\" -gt 0 ] && [ \"$length2\" -gt 0 ] \n then \n rm -rf $PRIMARY/* \n rm -rf $SECONDARY/* \n for el in $primary_partition \n do \n scp david@${LOCATION[0]}:$dir1/t1_weekly_1680_\"$el\"_200003_5.data $PRIMARY/. || scp david@${LOCATION[1]}:$dir2/t1_weekly_1680_\"$el\"_200003_5.data $PRIMARY/. \n done \n fi"}'

This is the error I am getting - 

ValueError: Invalid control character

Can anyone help me what wroing I am doing? And what's the right way to make above shell script in a Single line JSON document?

UPDATE:-

There is another twist into this which I thought its worth mentioning..

I need to store this single line shell script data in a Zookeeper node. So for Hello World shell script example case, I am storing by using the following code. I am using Curator library to write into Zookeeper. 

client.create().creatingParentsIfNeeded().forPath("/be/wf/error1/v1/step1", "{\"description\":\"Hello World 1.\", \"script\":\"#!/bin/bash \\n set -e \\n echo Hello World 1 \\n\"}".getBytes());

And then I have my Python program to read the Zookeeper node data and then execute the shell script by extracting from the script tag..

Now I tried representing the same script as mentioned in the answer, in the above format, and my Eclipse started giving compilation errors on the string. So how do I represent the shell script as given in the answer in the above..

I guess I really need a JSON so that I can store it properly in Zookeeper node and then my Python program can read the same JSON data from the node and execute the shell script from the script tag.

Upvotes: 1

Views: 180

Answers (1)

user2357112
user2357112

Reputation: 281486

First, consider whether you really need JSON. In your example code, you create a JSON string and then immediately decode it into a Python dict. Would it be simpler to just use a dict directly?

The problem with your current string is that you're not escaping your quotation marks properly. To avoid confusing multi-level escaping, use a triple-quoted string to represent the shell script and json.dumps to convert a dict into a JSON string:

import json
jsonstr = json.dumps({"script": """\
#!/bin/bash
set -e

readonly PRIMARY=/tech01/primary
readonly SECONDARY=/tech02/secondary
readonly LOCATION=(machineA machineB)
readonly MAPPED_LOCATION=/bat/data/snapshot
HOSTNAME=$hostname

dir1=$(ssh -o "StrictHostKeyChecking no" david@${LOCATION[0]} ls -dt1 "$MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)
dir2=$(ssh -o "StrictHostKeyChecking no" david@${LOCATION[1]} ls -dt1 "$MAPPED_LOCATION"/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9] | head -n1)

echo $dir1
echo $dir2

length1=$(ssh -o "StrictHostKeyChecking no" david@${LOCATION[0]} "ls '$dir1' | wc -l")
length2=$(ssh -o "StrictHostKeyChecking no" david@${LOCATION[1]} "ls '$dir2' | wc -l")

echo $length1
echo $length2

if [ "$dir1" = "$dir2" ] && [ "$length1" -gt 0 ] && [ "$length2" -gt 0 ]
then
    rm -rf $PRIMARY/*
    rm -rf $SECONDARY/*
    for el in $primary_partition
    do
        scp david@${LOCATION[0]}:$dir1/weekly_8880_"$el"_5.data $PRIMARY/. || scp david@${LOCATION[1]}:$dir2/weekly_8880_"$el"_5.data $PRIMARY/.
    done
fi"""})

Alternatively, you could put the shell script in its own file and open the file to read the string from it.

Upvotes: 3

Related Questions