CODEWITHSUNDEEP
pythondownloadwget
user2044983
user2044983

Reputation: 47

Download file using wget

I need to download fits files from a webpage. I'm doing this using -i wget options: I store the download file in a list.txt file which contains URL1, URL2... and then

$ wget -i list.txt

Do you know if there is the possibility to do the same thing using a Python script?

Upvotes: 0

Views: 2700

Answers (3)

Szczerski
Szczerski

Reputation: 1011

If you get SSL: CERTIFICATE_VERIFY_FAILED:

import wget
import ssl

ssl._create_default_https_context = ssl._create_unverified_context
with open('list.txt') as my_list:
    for url in my_list:
        wget.download(url)

Using the OS library:

import os

with open('list.txt') as my_list:
    for url in my_list:
        os.system('wget ' + url)

Upvotes: 1

Jaydev
Jaydev

Reputation: 1812

with open('list.txt') as my_list:
    for url in my_list:
        wget.download(url)

Upvotes: 0

mbatchkarov
mbatchkarov

Reputation: 16109

Assuming your file contains one URL per line, you can do this:

import urllib2
with open('list.txt') as my_list:
    for line in my_list:
        response = urllib2.urlopen(line)
        html = response.read()
        # now process the page's source

Upvotes: 1

Related Questions