CODEWITHSUNDEEP
bashoptimizationcurlsed
yab
yab

Reputation: 113

Properly using curl --data-urlencode when passing a variable

I'm trying to optimize my code and one particular piece of code is borrowed. I'd like to remove the sed so I'm not using any external processes in my main loop.

function sendMsg () {
value=$(echo $1 | sed 's/ /%20/g;s/!/%21/g;s/"/%22/g;s/#/%23/g;s/\&/%26/g;s/'\''/%28/g;s/(/%28/g;s/)/%29/g;s/:/%3A/g;s/\//%2F/g');
str="http://www.xxxx.com/api.ashx?v=1&k=$Key&a=send&w=$value";
curl -s $str;
}

I've edited this for clarity. The $value is simply to convert to a proper url for output via the curl command at the end of the function.

While this works just fine, I'm mostly interested in making this as fast to process as possible without forking to outside processes if I can.

Thanks for the comments so far!

Where I'm at so far is this:

function sendMsg () {
str="http://www.xxxx.com/api.ashx?v=1&k=$Key&a=send&w=";
curl -s $str --data-urlencode "$1";
}

Am I on the right track at least?

Upvotes: 9

Views: 41993

Answers (2)

David W.
David W.

Reputation: 107080

First, the answer to your question: If you are doing a single substitution or filtering, using pattern matching is faster:

$ foo=${bar/old/new}               # Faster
$ foo=$(sed 's/old/new/' <<<$bar   # Slower

The first doesn't require spawning a sub-shell, and running sed, then substituting this back into $foo. However, if you are doing this almost a dozen times, I believe using sed may be faster:

value=$(sed -e 's/ /%20/g' \
   -e 's/!/%21/g' \
   -e 's/"/%22/g' \
   -e 's/#/%23/g' \
   -e 's/\&/%26/g' \
   -e 's/'\''/%28/g' \
   -e 's/(/%28/g' \
   -e 's/)/%29/g' \
   -e 's/:/%3A/g' \
   -e 's/\//%2F/g'<<<$1);

Note that this syntax is easier to read since each substitution command is on its own line. Also note that <<< eliminates the need to echo and pipe.

This only does a single call to sed while pattern matching has to be done multiple times.

However, you should be using --data and --data-uuencode instead of building the query string yourself:

$ curl -s http://www.xxxx.com/api.ashx \
    --data v=1 \
    --data k=$Key \
    --data a=send \
    --data-urlencode w="$value";

The --data--urlencode will encode the value of $value for you, so you don't have to do it. Unfortunately, this parameter doesn't exist in all versions of curl. It was added in version 7.18.0 back in January of 2008. Run curl --version to see what version you have:

$ curl --version     # Life is good
curl 7.30.0 (x86_64-apple-darwin13.0) libcurl/7.30.0 SecureTransport zlib/1.2.5

$ curl --version     # David Sad
curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5

Addendum

In attempting this I'm getting an 'unsupported API versionERROR', even though my curl --version reports 7.29.0

I can't test what you have, but I decided to try our Jenkins server to see if I can set the build description. I made sure the description has spaces in it, so it requires --data-urlencoding. This command worked:

$ curl --user dweintraub:swordfish \
    --data Submit=Submit \
    --data-urlencode description="This is my test descripition" \
    http://jenkins.corpwad.com/jenkins/job/Admin-5.1.1/138/submitDescription

This is as if I did:

$ curl -user "dweintraub:swordfish http://jenkins.corpwad.com/jenkins/job/Admin-5.1.1/138/submitDescription?Submit=Submit&desciption=This%20is%20my%20test%20descripition"

Note that --data adds the question mark for you.

(No, swordfish isn't my password).

It's not as complex as your command, but it might help point out where you're having problems. Do you have a user name and password? If so, you need the --user parameter.

Upvotes: 9

chepner
chepner

Reputation: 532112

If you have to do this a lot, the overhead of spawning multiple sed processes might add up. In that case, you can use the following lines instead:

value=${1// /%20/}
value=${value//!/%21}
value=${value//\"/%22}
value=${value//\#/%23}
value=${value//&/%26}
value=${value//\'/%27}
value=${value//(/%28}
value=${value//)/%29}
value=${value//:/%3A}
value=${value//\//%2F}

Upvotes: 3

Related Questions