CODEWITHSUNDEEP
bashcurl
David Dennis
David Dennis

Reputation: 722

Using curl with an unpredictable target filename

The filename of my curl download target is unpredictable and globbing with an asterisk isn't possible. I can download the file using the following command, but only after I've determined its' name in advance:

curl -O -vvv -k -u user:password https://myURL/ws/myfile.zip

How can I tailor my curl command to succeed with an unpredictable target name?

Upvotes: 4

Views: 29978

Answers (2)

F1Linux
F1Linux

Reputation: 4373

Intro:

Like the OP, I had a similar issue scripting the download of a binary- for docker-compose- from Github because the version number keeps iterating making the file name unpredictable.

This is how I solved it. Might not be the tidiest solution, but if you have a more elegant way, ping me a comment and I'll update the answer.

Solution:

I merely used an auto-populating variable that takes the output of curl, prints the 1st line- which will be the most recent release- and thengrep for the release number prefaced by a "v". The result is saved to the the path /home/ubuntu as the arbitrary file name "docker-compose-latest"

curl -L "https://github.com/docker/compose/releases/download/$(curl https://github.com/docker/compose/releases | grep -m1 '<a href="/docker/compose/releases/download/' | grep -o 'v[0-9:].[0-9].[0-9]')/docker-compose-$(uname -s)-$(uname -m)" -o /home/ubuntu/docker-compose-latest

And we validate that we received the correct binary (I'm downloading to a Raspberry Pi which has an ARM processor on 64 bit Ubuntu 20.04 LTS:

file /home/ubuntu/docker-compose-latest

Produces the following feedback on the file:

/home/ubuntu/docker-compose-latest: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=QqyJMzYMWOofWehXt3pb/T7U4zg-t8Xqz_11RybNZ/ukJOlZCpzQuZzBcwSK3b/d6ecQ2m2VfqKb_EQRUZA, stripped

To validate this solution works, just execute the above commands remembering to change the path of the file command if not using Ubuntu.

Conclusion:

Again, might not be the most elegant solution, but it's a solution for how one can download a target with curl that has an unpredictable filename.

Upvotes: 1

John Moon
John Moon

Reputation: 924

There's no easy way to get a directory listing using HTTP. You can use curl to just print the HTML generated by the site. If there's an index with links to the files on that server, simply running

curl -s -u user:password https://myURL/ws/ | grep .zip

will print HTML-formatted links to the zip files available for download on that page.

Upvotes: 5

Related Questions