CODEWITHSUNDEEP
python
John Smith
John Smith

Reputation: 363

Python: Scan all sites of a website for specific URLs

I want to scan my forum for specific links. All links look like this: http://www.vbulletinxyz-forum.tld/forum/showthread.php?t=17590. Only the thread-number at the end of the link changes.

Currently I am using the following code, but it only works for one specific URL, not all threads of the forum. How would I have to change the code to let it scan all threads?

import urllib
mypath = "http://vbulletin-forumxyz.tld/forum/showthread.php?t=1"
mylines = urllib.urlopen(mypath).readlines()
for item in mylines:
    if "http://specific.tld" in item:
        print item[item.index("http://specific.tld"):]

Upvotes: 0

Views: 707

Answers (2)

John Smith
John Smith

Reputation: 363

This is how it works and checks threads from 0 to 400,000.

import urllib.request
import time
import codecs

def mypath(t):
    return "http://www.someforum.org/forum/showthread.php?t={}".format(t)


for t in range(0,400000):
    conn = urllib.request.urlopen(mypath(t))

    # check status code
    if conn.getcode() != 200:
        continue

    mylines = conn.read().decode('windows-1251').splitlines()

    for item in mylines:
        if "http://someurl.tld" in item:
            print(item)

   # avoid fetching to fast (you might get banned otherwise)
   # time.sleep(0.5)

Upvotes: 0

Raphael
Raphael

Reputation: 1801

  1. either by trying all thread numbers
  2. or by using a spider which follows links (and discovers new threads)

(1) is easy to implement but probably not all thread numbers (t) are existent. So there will be a lot of 404 requests.

(2) take a look at scrapy

update (1): here is how it can be done in principle. Note that a) the url you provided is not reachable (dummy) so i did not test it and b) its python 3.X

import urllib.request
import time


def mypath(t):
    return "http://vbulletin-forumxyz.tld/forum/showthread.php?t={}".format(t)


for t in range(2):
    conn = urllib.request.urlopen(mypath(t))

    # check status code
    if conn.getcode() != 200:
        continue

    mylines = conn.read().decode('utf-8').splitlines()
    for item in mylines:
        if "http://specific.tld" in item:
            print(item)

   # avoid fetching to fast (you might get banned otherwise)
    time.sleep(0.5)

Upvotes: 1

Related Questions