CODEWITHSUNDEEP
bashcurl
Mare
Mare

Reputation: 963

Bash curl and variable in the middle of the url

I would need to read certain data using curl. I'm basically reading keywords from file

while read line
do
    curl 'https://gdata.youtube.com/feeds/api/users/'"${line}"'/subscriptions?v=2&alt=json' \
         > '/home/user/archive/'"$line"
done < textfile.txt

Anyway I haven't found a way to form the url to curl so it would work. I've tried like every possible single and double quoted versions. I've tried basically:

'...'"$line"'...'
"..."${line}"..."
'...'$line'...'

and so on.. Just name it and I'm pretty sure that I've tried it.

When I'm printing out the URL in the best case it will be formed as:

 /subscriptions?v=2&alt=jsoneeds/api/users/KEYWORD FROM FILE

or something similar. If you know what could be the cause of this I would appreciate the information. Thanks!

Upvotes: 30

Views: 127430

Answers (4)

Gordon Davisson
Gordon Davisson

Reputation: 125808

It's not a quoting issue. The problem is that your keyword file is in DOS format -- that is, each line ends with carriage return & linefeed (\r\n) rather than just linefeed (\n). The carriage return is getting read into the line variable, and included in the URL. The giveaway is that when you echo it, it appears to print:

/subscriptions?v=2&alt=jsoneeds/api/users/KEYWORD FROM FILE"

but it's really printing:

https://gdata.youtube.com/feeds/api/users/KEYWORD FROM FILE
/subscriptions?v=2&alt=json

...with just a carriage return between them, so the second overwrites the first.

So what can you do about it? Here's a fairly easy way to trim the cr at the end of the line:

cr=$'\r'
while read line
do
    line="${line%$cr}"
    curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" \
         > "/home/user/archive/$line"
done < textfile.txt

Upvotes: 41

Mattias Ahnberg
Mattias Ahnberg

Reputation: 2870

Just use it like this, should be sufficient enough:

curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" > "/home/user/archive/${line}"

If your shell gives you issues with & just put \&, but it works fine for me without it.

Upvotes: 9

Jonathan Leffler
Jonathan Leffler

Reputation: 753970

If the data from the file can contain spaces and you have no objection to spaces in the file name in the /home/user/archive directory, then what you've got should be OK.

Given the contents of the rest of the URL, you could even just write:

while read line
do
    curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" \
         > "/home/user/archive/${line}"
done < textfile.txt

where strictly the ${line} could be just $line in both places. This works because the strings are fixed and don't contain shell metacharacters.

Since you're code is close to this, but you claim that you're seeing the keywords from the file in the wrong place, maybe a little rewriting for ease of debugging is in order:

while read line
do
    url="https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json"
    file="/home/user/archive/${line}"
    curl "$url" > "$file"
done < textfile.txt

Since the strings may end up containing spaces, it seems (do you need to expand spaces to + in the URL?), the quotes around the variables are strongly recommended. You can now run the script with sh -x (or add a line set -x to the script) and see what the shell thinks it is doing as it is doing it.

Upvotes: 1

Fred Foo
Fred Foo

Reputation: 363607

Your current version should work, I think. More elegant is to use a single pair of double quotes around the whole URL with the variable in ${}:

"https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json"

Upvotes: 28

Related Questions