CODEWITHSUNDEEP
bashcurlparallel-processing
Mindaugas Bernatavičius
Mindaugas Bernatavičius

Reputation: 3909

How to correctly pass arguments to curl using GNU parallel in BASH

I have a similar query to ElasticSearch:

printf "${DATES[@]}" | \
perl -pne 's/_/\n/g' | \
parallel --jobs $i --load 6 curl -s -u user:pass \
         -XPOST 'https://127.0.0.1/_search' \
         -d "{\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"query\":\"_type:some_logs AND webapp_domain:${DOMAIN}\",\"analyze_wildcard\":false}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":\"{}T00:00:00.001Z\",\"lte\":\"{}T23:59:59.999Z\"}}}],\"must_not\":[]}}}},\"size\":1}"

I'm having prodlems with the JSON that is being passed:

if I do (double qoutes):

-d "query"

I get:

 + parallel --jobs 4 --load 6 curl -s -u user:pass -XPOST https://127.0.0.1/_search -d '{"query":{"filtered":{"query":{"query_string":{"query":"_type:some_logs AND webapp_domain:www.some.tk","analyze_wildcard":false}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"gte":"{}T00:00:00.001Z","lte":"{}T23:59:59.999Z"}}}],"must_not":[]}}}},"size":1}'
{"error":"SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[3YqwKjGDTaKREWzQdxIjzA][%{index_name}][0]: RemoteTransportException[[elasticsearch5][inet[/1.1.1.1:9300]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[%{index_name}][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [query:{filtered:query:{query_string:query:_type:some_logs AND webapp_domain:www.some.tk}}]]]; nested: JsonParseException[Unrecognized token 'query': was expecting ('true', 'false' or 'null')\n at [Source: UNKNOWN; line: 1, column: 7]]; }{[3YqwKjGDTaKREWzQdxIjzA][.jsp][0]: RemoteTransportException[[elasticsearch5][inet[/1.1.1.1:9300]][indices:data/read/search[phase/query]]]; nested: SearchParseException[[.jsp][0]: from[-1],size[-1]: Parse Failure [...

If I do (single qoutes):

-d 'query'

I get:

/bin/bash: -c: line 0: `curl -s -u user:pass -XPOST https://172.0.0.1/_search -d {\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"query\":\"_type:some_logs AND webapp_domain:${DOMAIN}\",\"analyze_wildcard\":false}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":\"2016-03-22T00:00:00.001Z\",\"lte\":\"2016-03-22T23:59:59.999Z\"}}}],\"must_not\":[]}}}},\"size\":1}'
/bin/bash: -c: line 0: syntax error near unexpected token `('

So the DATES are being populated correctly, but the query does not interpolate the ${DOMAIN} parameter. This is expected as the '' instruct the interpreter to treat the sting literally (no interpolation).

QUESTION:

How can and whether can I do it? Can I pass the ${DATES[@]} 1 at a time using parallels and pass the ${DOMAIN} from the surrounding scope in the same call?

UPDATE (2016-03-29) #1:

I had some correctly working curls to ES from BASH - so I provoked the same error to check how the query looks from ES perspective. The results:

Initial syntax of the query from ES logs:

[Failed to parse source [query:{filtered:query:{query_string:query:_type:some_logs AND webapp_domain:www.some.tk}}]]];

This is how the qoutery looks when the statement is executed correctly:

[Failed to parse source [{\"query\":{\"filtered\":{\"query\":

I see that there are critical signs missing from the query.

Upvotes: 1

Views: 442

Answers (1)

Ole Tange
Ole Tange

Reputation: 33685

Quoting is a bitch. Quoting double is a double bitch.

Therefore the manual for GNU Parallel suggest making a function and running that instead. The benefit is that you can test the function separately before giving it to GNU Parallel.

In your case it will look something like:

mycurl() {
  time="$1"
  curl -s -u user:pass \
         -XPOST 'https://127.0.0.1/_search' \
         -d "{\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"query\":\"_type:some_logs AND webapp_domain:${DOMAIN}\",\"analyze_wildcard\":false}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":\"${time}T00:00:00.001Z\",\"lte\":\"${time}T23:59:59.999Z\"}}}],\"must_not\":[]}}}},\"size\":1}"
}
export -f mycurl

printf "${DATES[@]}" | \
perl -pne 's/_/\n/g' | \
parallel --jobs $i --load 6 mycurl

PS: Depending on $DATES you may be able to use GNU Parallel's -d to separate them insted of the perl filter.

Upvotes: 2

Related Questions