CODEWITHSUNDEEP
bashsed
Bastien974
Bastien974

Reputation: 321

Sed variable too long

I need to substitute a unique string in a json file: {FILES} by a bash variable that contains thousands of paths: ${FILES}

sed -i "s|{FILES}|$FILES|" ./myFile.json

What would be the most elegant way to achieve that ? The content of ${FILES} is a result of an "aws s3" command. The content would look like :

FILES="/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."

I can't think of a solution where xargs would help me.

Upvotes: 2

Views: 980

Answers (5)

Ruud Helderman
Ruud Helderman

Reputation: 11018

The safest way is probably to let Bash itself expand the variable. You can create a Bash script containing a here document with the full contents of myFile.json, with the placeholder {FILES} replaced by a reference to the variable $FILES (not the contents itself). Execution of this script would generate the output you seek.

For example, if myFile.json would contain:

{foo: 1, bar: "{FILES}"}

then the script should be:

#!/bin/bash
cat << EOF
{foo: 1, bar: "$FILES"}
EOF

You can generate the script with a single sed command:

sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json

Notice sed is doing two replacements; the first one (s/\$/\\$/g) to escape any dollar signs that might occur within the JSON data (replace every $ by \$). The second replaces {FILES} by $FILES; the literal text $FILES, not the contents of the variable.

Now we can combine everything into a single Bash one-liner that generates the script and immediately executes it by piping it to Bash:

sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | /bin/bash

Or even better, execute the script without spawning a subshell (useful if $FILES is set without export):

sed -e '1i#!/bin/bash\ncat << EOF' -e 's/\$/\\$/g;s/{FILES}/$FILES/' -e '$aEOF' myFile.json | source /dev/stdin

Output:

{foo: 1, bar: "/file1.ipk, /file2.ipk, /subfolder1/file3.ipk, /subfolder2/file4.ipk, ..."}

Upvotes: 2

Peter Bowers
Peter Bowers

Reputation: 3093

Instead of putting all those variables in an environment variable, put them in a file. Then read that file in perl:

foo.pl:

open X, "$ARGV[0]" or die "couldn't open";
shift;
$foo = <X>;
while (<>) {
   s/world/$foo/;
   print;
}

Command to run:

aws s3 ... >/tmp/myfile.$$
perl foo.pl /tmp/myfile.$$ <myFile.json >newFile.json

Hopefully that will bypass the limitations of the environment variable space and the argument length by pulling all the processing within perl itself.

Upvotes: 0

Barett
Barett

Reputation: 5948

Maybe just don't do it? Can you just :

echo "var f = " > myFile2.json
echo $FILES >> myFile2.json

And reference myFile2.json from within your other json file? (You should put the global f variable into a namespace if this works for you.)

Upvotes: 0

Peter Bowers
Peter Bowers

Reputation: 3093

It's a little gross, but you can do it all within shell...

while read l
do
    if ! echo "$l" | grep -q '{DATA}'
    then
        echo "$l"
    else
        echo "$l" | sed 's/{DATA}.*$//'
        echo "$FILES"
        echo "$l" | sed 's/^.*{DATA}//'
    fi
done <./myfile.json >newfile.json
#mv newfile.json myfile.json

Obviously I'd leave the final line commented until you were confident it worked...

Upvotes: 0

Peter Bowers
Peter Bowers

Reputation: 3093

Maybe perl would have fewer limitations?

perl -pi -e "s#{FILES}#${FILES}#" ./myFile.json

Upvotes: 0

Related Questions