CODEWITHSUNDEEP
bashvariablesawkarguments
Polizi8
Polizi8

Reputation: 277

Awk argument list too long, but only if I increase the data in the variable

I wrote a working bash script that loops over some files to read their plain text, modify it with sed and replace a string in a template file with the modified text.
I achieved this by getting rid of sed for the replace part (I found quite difficult using it for multiline substitution in my specific case).
awk worked fine until now, when I'm trying to increase the number of paragraph in a file (from 200 to 225). I'm getting the error script.sh: line 11: /usr/bin/awk: Argument list too long, and the template file is now chopped way before (Opening, 14 paragraphs, Closing) than when it was working (Opening, 200 paragraphs, Closing).

Why does this happen? How can I solve it, possibly keeping awk?
Why does awk print only 14 paragraphs now, instead of the 200 it could print before?
Why doesn't set -e stop the script after that error appears?

This it the script that works until a file becomes too big:

#!/bin/bash
set -e

cp template.bkp template.txt

for file in text/* ; do
    modifiedText=$(sed '...' $file | fold -w 50 -s)
    modifiedText+="

#REPLACESTRING"
    awk -v modifiedText="$modifiedText" '{gsub("#REPLACESTRING", modifiedText, $0); print}' template.txt > template-tmp.txt && mv template-tmp.txt template.txt
done
awk '{gsub("#REPLACESTRING", "", $0); print}' template.txt > template-tmp.txt && mv template-tmp.txt template.txt

[...]

The template file looks like this:

Opening

#REPLACESTRING

Closing

Upvotes: 0

Views: 392

Answers (1)

karakfa
karakfa

Reputation: 67507

After a quick read it looks like you're appending a number of files and adding a header and footer. If so, it's better to revert the operations

tempfile=$(mktemp)
sed '...' text/* | fold -w 50 -s >> "$tempfile"    
sed -e '/#REPLACEMENT/ {' -e "r $tempfile" -e 'd' -e '}' template > output
rm "$tempfile"

sed can operate on multiple files, no need to loop through. Also use bash as the orchestration of operations and keep text in files and do text processing with tools like awk and sed, which you did partially.

Upvotes: 2

Related Questions