Download files by script or something

Question

In a web site, not mine, there a result to a search

<a href="show?file=191719&token=r1j">
<a href="show?file=191720&token=gh5">
<a href="show?file=191721&token=98j">
.....
<a href="show?file=191733&token=ty0">

and after I click on one of them I go to a page i fill a form and after I go to download page and i click on the link:

<a href="download?file=191719&token=r1j">

And i have to do that manually for 150 file wich is too long !!

what i want is by using a script or something, i download all the files directly by getting the file id in result page and put it in download link.

Answer 1

You could use an XMLHttpRequest to download files in parallel as blobs and then use <a download>s to initiate download behaviour. This will have same-origin-policy restrictions though.

General idea is

// fetch
var xhr = new XMLHttpRequest();
xhr.addEventListener('load', function () {
    var uri = URL.createObjectURL(this.response); // generate URI to access Blob
    // write, see below
});
xhr.open('GET', target_file_href);
xhr.responseType = 'blob'; // state we want the target as a blob/file
xhr.send(); // send the request
// ---------------

// write
var a = document.createElement('a');
a.href = uri;
a.setAttribute('download'); // make this a download link rather than a change page
document.body.appendChild(a);
a.click();
// cleanup a, uri

Here is a parallel file downloader I wrote in ES5 which limits the number of concurrent downloads.

function ParallelDownloader(max_parallel, retry_on_error) {
    this.links = [];
    this.current = 0;
    this.max_parallel = max_parallel || 5;
    this.retry_on_error = !!retry_on_error;
}
ParallelDownloader.prototype = Object.create(null);
ParallelDownloader.prototype.add = function (url) {
    if ('splice' in url && 'length' in url)
        this.links.push.apply(this.links, url);
    else
        this.links.push(url);
    this.downloadNext();
};
ParallelDownloader.prototype.downloadNext = (function () {
    function load() {
        var a = document.createElement('a'),
            uri = URL.createObjectURL(this.response),
            cd = this.getResponseHeader('Content-Disposition'),
            filename = null;
        if (cd) {
            cd = cd.match(/;\s+filename=(.+)/);
            if (cd) filename = cd[1];
        }
        if (null === filename) {
            cd = this.__url.match(/\/([^/]+?(?=\?|$))/);
            if (cd) filename = cd[1];
        }
        if (null !== filename) a.setAttribute('download', filename);
        else a.setAttribute('download');
        a.setAttribute('href', uri);
        document.body.appendChild(a);
        a.click();
        document.body.removeChild(a);
        URL.revokeObjectURL(uri);
        --this.__parallelDownloader.current;
        this.__parallelDownloader.downloadNext();
    }
    function error() {
        --this.__parallelDownloader.current;
        if (this.__parallelDownloader.retry_on_error) {
            console.warn('Will retry', this.__url);
            this.__parallelDownloader.unshift(this.__url);
        }
        this.__parallelDownloader.downloadNext();
    }
    return function () {
        var url;
        ++this.current;
        if (this.current > this.max_parallel || this.links.length === 0) {
            --this.current;
            return;
        }
        url = this.links.shift();
        var xhr = new XMLHttpRequest();
        xhr.__parallelDownloader = this;
        xhr.__url = url;
        xhr.addEventListener('load', load);
        xhr.addEventListener('error', error);
        xhr.open('GET', url);
        xhr.responseType = 'blob';
        xhr.send();
        this.downloadNext();
    };
}());

To use it you would do, e.g.

var pd = new ParallelDownloader(10); // max 10 concurrent downloads
pd.add([
    '/path1.txt', '/path2.pub', '/path3.pdf'
]);
// or
pd.add('/path4.txt');
pd.add('/path5.txt');
// etc

Download attempt initiates as soon as a link is added and there is a slot free. (If you enable retry_on_error I haven't limited it so you may get infinite loops)

Answer 2

use this javascript snippet, where http://www.that-website.com/ is the url of that website, AND DO NOT download all files all at once if there are too many, download couple dozens each time by specifying start and finish file number, Note that the browser popup blocker will block this so you need to allow popup from this webpage in your popup blocker in your browser

JS:

var fileNumber,
start = 191719,
finish = 191729;
for(fileNumber = start; fileNumber <= finish; ++fileNumber){
    window.open("http://www.that-website.com/download?file=" + fileNumber);
}

UPDATE: Since random token are implemented in the url the easiest way is to enter it manually in multi-lines of window.open(), something like this:

window.open("http://www.that-website.com/download?file=191719&token=r1j");
window.open("http://www.that-website.com/download?file=191720&token=gh5");
window.open("http://www.that-website.com/download?file=191721&token=98j");

and so on for couple dozens.

UPDATE 2: See an example of this in this JSFiddle

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <title>Document</title>
    </head>
    <body>
        <!-- COPY BUNCH OF THE URLs AND PASTE THEM IN HERE THEN RELOAD THE PAGE, THEN REPEAT OVER AND OVER UNTIL IT IS ALL DONE! -->
        <script src="https://code.jquery.com/jquery-1.11.3.min.js"></script>
        <script>
            $(document).ready(function(){
                $('a').each(function(){
                    var showLink = $(this).attr('href');
                    var downloadLink = showLink.replace("show?file", "download?file");
                    window.open("http://www.example.com/" + downloadLink);
                });
            });
        </script>
    </body>
</html>

With the above code, this an HTML page ON YOUR COMPUTER, copy several original from that website page links - like: <a href="http://www.example.com/show?file=111&token=23f">TEST</a> to your local page and run it, still it is highly recommended that you paste 10-30 links each time.

Answer 3

You can generate links using excel, save it as txt file and download using wget with -i parameter.