CODEWITHSUNDEEP
regexlinuxbashwget
code4fun
code4fun

Reputation: 2749

How to download an entire directory and subdirectories using wget?

I am trying to download the files for a project using wget, as the SVN server for that project isn't running anymore and I am only able to access the files through a browser. The base URLs for all the files is the same like

http://abc.tamu.edu/projects/tzivi/repository/revisions/2/raw/tzivi/*

How can I use wget (or any other similar tool) to download all the files in this repository, where the "tzivi" folder is the root folder and there are several files and sub-folders (upto 2 or 3 levels) under it?

Upvotes: 182

Views: 361891

Answers (9)

gti3993
gti3993

Reputation: 1

This will help

wget -m -np -c --level 0 --no-check-certificate -R"index.html*"http://www.your-websitepage.com/dir

Upvotes: -1

Android Cse
Android Cse

Reputation: 51

try this working code (30-08-2021):

!wget --no-clobber --convert-links --random-wait -r -p --level 1 -E -e robots=off --adjust-extension -U mozilla "yourweb directory with in quotations"

Upvotes: 0

Nelinton Medeiros
Nelinton Medeiros

Reputation: 111

This link just gave me the best answer:

$ wget --no-clobber --convert-links --random-wait -r -p --level 1 -E -e robots=off -U mozilla http://base.site/dir/

Worked like a charm.

Upvotes: 11

Sarkar_lat_2016
Sarkar_lat_2016

Reputation: 51

wget -r --no-parent URL --user=username --password=password

the last two options are optional if you have the username and password for downloading, otherwise no need to use them.

You can also see more options in the link https://www.howtogeek.com/281663/how-to-use-wget-the-ultimate-command-line-downloading-tool/

Upvotes: 5

Hiep Luong
Hiep Luong

Reputation: 1

This works:

wget -m -np -c --no-check-certificate -R "index.html*" "https://the-eye.eu/public/AudioBooks/Edgar%20Allan%20Poe%20-%2"

Upvotes: -1

user2936450
user2936450

Reputation:

You may use this in shell:

wget -r --no-parent http://abc.tamu.edu/projects/tzivi/repository/revisions/2/raw/tzivi/

The Parameters are:

-r     //recursive Download

and 

--no-parent // Don´t download something from the parent directory

If you don't want to download the entire content, you may use: 

-l1 just download the directory (tzivi in your case)

-l2 download the directory and all level 1 subfolders ('tzivi/something' but not 'tivizi/somthing/foo')

And so on. If you insert no -l option, wget will use -l 5 automatically.

If you insert a -l 0 you´ll download the whole Internet, because wget will follow every link it finds.

Upvotes: 258

Rajiv Yadav
Rajiv Yadav

Reputation: 271

You can use this in a shell:

wget -r -nH --cut-dirs=7 --reject="index.html*" \
      http://abc.tamu.edu/projects/tzivi/repository/revisions/2/raw/tzivi/

The Parameters are:

-r recursively download

-nH (--no-host-directories) cuts out hostname 

--cut-dirs=X (cuts out X directories)

Upvotes: 27

baobab33
baobab33

Reputation: 21

you can also use this command :

wget --mirror -pc --convert-links -P ./your-local-dir/ http://www.your-website.com

so that you get the exact mirror of the website you want to download

Upvotes: 2

lanni654321
lanni654321

Reputation: 1087

use the command

wget -m www.ilanni.com/nexus/content/

Upvotes: 5

Related Questions