CODEWITHSUNDEEP
pythonhtmlweb-scraping
Hritik Kumar
Hritik Kumar

Reputation: 23

How to scrape pdf links from webpages having unchanging urls?

I am working on a project on web scraping and I am asked to scrape all the pdf links from a website: https://www.sebi.gov.in/sebiweb/home/HomeAction.do?doListing=yes&sid=3&s . The website has 397 pages but every page has the same URL. I tried the inspect element tool and found out that a javascript code helps to navigate to different pages. But still I am not able to figure out how to run my script for all the pages.

Below is my code.

from bs4 import BeautifulSoup
import lxml
url = 'https://www.sebi.gov.in/sebiweb/home/HomeAction.do?doListing=yes&sid=3&s'

conn = urllib2.urlopen(url)
html = conn.read()

soup = BeautifulSoup(html)
links = soup.find_all('a')
urls=[]
for tag in links:
    link = tag.get('href',None)
    if link is not None and link.endswith('html'):
        #urls.append(link)
        purl=link
        new=urllib2.urlopen(purl)
        htm=new.read()
        sp=BeautifulSoup(htm)
        nl=sp.find_all('a')
        nm=sp.find_all('iframe')
        for i in nl:
            q=i.get('href',None)
            title=i.get('title',None)
            if q is not None and q.endswith('pdf'):
                print(q)
                urls.append(q)
        for j in nm:
            z=j.get('src',None)
            title=j.get('title',None)
            if z is not None and z.endswith('pdf')and title is not None:
                print(z)
                print(title)
                urls.append(z)
        

print(len(urls))

Upvotes: 2

Views: 232

Answers (2)

Andrej Kesely
Andrej Kesely

Reputation: 195438

You can use their API located on https://www.sebi.gov.in/sebiweb/ajax/home/getnewslistinfo.jsp to load the data.

For example:

from bs4 import BeautifulSoup
from requests import get


api_url = 'https://www.sebi.gov.in/sebiweb/ajax/home/getnewslistinfo.jsp'
payload = {
    'nextValue': "1",
    'next': "n",
    'search': "",
    'fromDate': "",
    'toDate': "",
    'fromYear': "",
    'toYear': "",
    'deptId': "",
    'sid': "3",
    'ssid': "-1",
    'smid': "0",
    'intmid': "-1",
    'sText': "Filings",
    'ssText': "-- All Sub Section --",
    'smText': "",
    'doDirect':  "1",
}

page = 0
while True:
    print('Page {}...'.format(page))
    payload['doDirect'] = page
    soup = BeautifulSoup(requests.post(api_url, data=payload).content, 'html.parser')
    rows = soup.select('tr:has(td)')
    if not rows:
        break
    for tr in rows:
        row = [td.get_text(strip=True) for td in tr.select('td')] + [tr.a['href']]
        print(*row, sep='\t')
    page += 1

Prints:

...

Page 1...
Jun 25, 2020    Mindspace Business Parks REIT – Addendum to Draft Prospectus    https://www.sebi.gov.in/filings/reit-issues/jun-2020/mindspace-business-parks-reit-addendum-to-draft-prospectus_46928.html
Jun 25, 2020    Amrit Corp. Ltd. - Public Announcement  https://www.sebi.gov.in/filings/buybacks/jun-2020/amrit-corp-ltd-public-announcement_46927.html
Jun 24, 2020    NIIT Technologies Buyback - Post Buyback - Public Advertisement https://www.sebi.gov.in/filings/buybacks/jun-2020/niit-technologies-buyback-post-buyback-public-advertisement_46923.html
Jun 23, 2020    Addendum to Letter of Offer of Arvind Fashions Limited  https://www.sebi.gov.in/filings/rights-issues/jun-2020/addendum-to-letter-of-offer-of-arvind-fashions-limited_46941.html
Jun 23, 2020    Genesis Exports Limited - Draft letter of Offer https://www.sebi.gov.in/filings/buybacks/jun-2020/genesis-exports-limited-draft-letter-of-offer_46911.html
Jun 23, 2020    Genesis Exports Limited - Public Announcement   https://www.sebi.gov.in/filings/buybacks/jun-2020/genesis-exports-limited-public-announcement_46909.html
Jun 19, 2020    Coral India Finance and Housing Limited – Post Buy-back Public Announcement https://www.sebi.gov.in/filings/buybacks/jun-2020/coral-india-finance-and-housing-limited-post-buy-back-public-announcement_46900.html
Jun 19, 2020    Network Limited https://www.sebi.gov.in/filings/takeovers/jun-2020/network-limited_46890.html
Jun 17, 2020    KSOLVES INDIA LIMITED   https://www.sebi.gov.in/filings/public-issues/jun-2020/ksolves-india-limited_46996.html
Jun 10, 2020    Happiest Minds Technologies Limited https://www.sebi.gov.in/filings/public-issues/jun-2020/happiest-minds-technologies-limited_46843.html
Jun 08, 2020    IM+ Capitals Limited    https://www.sebi.gov.in/filings/takeovers/jun-2020/im-capitals-limited_46786.html
Jun 05, 2020    HealthCare Global Enterprises Limited   https://www.sebi.gov.in/filings/takeovers/jun-2020/healthcare-global-enterprises-limited_46773.html
Jun 02, 2020    Jaikumar Constructions Ltd. - DRHP  https://www.sebi.gov.in/filings/public-issues/jun-2020/jaikumar-constructions-ltd-drhp_46774.html
Jun 02, 2020    Mahindra Focused Equity Yojana  https://www.sebi.gov.in/filings/mutual-funds/jun-2020/mahindra-focused-equity-yojana_46767.html
Jun 02, 2020    GRANULES INDIA LIMITED - Dispatch advertisement https://www.sebi.gov.in/filings/buybacks/jun-2020/granules-india-limited-dispatch-advertisement_46765.html
Jun 02, 2020    GRANULES INDIA LIMITED  - Letter of Offer   https://www.sebi.gov.in/filings/buybacks/jun-2020/granules-india-limited-letter-of-offer_46764.html
Jun 02, 2020    Motilal Oswal Multi Asset Fund  https://www.sebi.gov.in/filings/mutual-funds/jun-2020/motilal-oswal-multi-asset-fund_46762.html
Jun 02, 2020    Principal Large Cap Fund    https://www.sebi.gov.in/filings/mutual-funds/jun-2020/principal-large-cap-fund_46761.html
Jun 02, 2020    Mahindra Arbitrage Yojana   https://www.sebi.gov.in/filings/mutual-funds/jun-2020/mahindra-arbitrage-yojana_46760.html
Jun 02, 2020    HSBC Mid Cap Equity Fund    https://www.sebi.gov.in/filings/mutual-funds/jun-2020/hsbc-mid-cap-equity-fund_46759.html
Jun 01, 2020    Tanla Solutions Limited - DLOF  https://www.sebi.gov.in/filings/buybacks/jun-2020/tanla-solutions-limited-dlof_46750.html
Jun 01, 2020    Axis Banking ETF    https://www.sebi.gov.in/filings/mutual-funds/jun-2020/axis-banking-etf_46748.html
Jun 01, 2020    Kalpataru Power Transmission Limited - Public Announcement  https://www.sebi.gov.in/filings/buybacks/jun-2020/kalpataru-power-transmission-limited-public-announcement_46746.html
Jun 01, 2020    Reliance Industries Limited - Addendum dated May 22, 2020   https://www.sebi.gov.in/filings/rights-issues/jun-2020/reliance-industries-limited-addendum-dated-may-22-2020_46745.html
Jun 01, 2020    Reliance Industries Limited - Addendum dated May 19, 2020   https://www.sebi.gov.in/filings/rights-issues/jun-2020/reliance-industries-limited-addendum-dated-may-19-2020_46744.html
Page 2...
Jun 01, 2020    Reliance Industries Limited - Addendum dated May 18, 2020   https://www.sebi.gov.in/filings/rights-issues/jun-2020/reliance-industries-limited-addendum-dated-may-18-2020_46743.html
May 29, 2020    Muthoottu Mini Financiers Limited- Prospectus   https://www.sebi.gov.in/filings/debt-offer-document/may-2020/muthoottu-mini-financiers-limited-prospectus_46769.html
May 29, 2020    Coral India Housing and Finance Limited - Advertisement https://www.sebi.gov.in/filings/buybacks/may-2020/coral-india-housing-and-finance-limited-advertisement_46732.html
May 29, 2020    TANLA SOLUTIONS LIMITED - Public Announcement   https://www.sebi.gov.in/filings/buybacks/may-2020/tanla-solutions-limited-public-announcement_46731.html
May 28, 2020    Tips Industries Limited - Dispatch Advertisement    https://www.sebi.gov.in/filings/buybacks/may-2020/tips-industries-limited-dispatch-advertisement_46723.html
May 27, 2020    KLM Axiva Finvest Limited - Prospectus  https://www.sebi.gov.in/filings/debt-offer-document/may-2020/klm-axiva-finvest-limited-prospectus_46755.html
May 26, 2020    Tips Industries Limited - Letter of Offer   https://www.sebi.gov.in/filings/buybacks/may-2020/tips-industries-limited-letter-of-offer_46708.html
May 26, 2020    Axis Capital Protection Oriented Fund - Series 7-10 https://www.sebi.gov.in/filings/mutual-funds/may-2020/axis-capital-protection-oriented-fund-series-7-10_46707.html
May 26, 2020    ICICI Prudential Alpha Low Vol 30 ETF   https://www.sebi.gov.in/filings/mutual-funds/may-2020/icici-prudential-alpha-low-vol-30-etf_46706.html
May 22, 2020    NIIT Technologies Ltd. - Letter of Offer    https://www.sebi.gov.in/filings/buybacks/may-2020/niit-technologies-ltd-letter-of-offer_46700.html
May 22, 2020    NIIT Technologies Ltd. - Dispatch Advertisement https://www.sebi.gov.in/filings/buybacks/may-2020/niit-technologies-ltd-dispatch-advertisement_46699.html
May 22, 2020    Coral India Finance and Housing Limited - Letter of Offer   https://www.sebi.gov.in/filings/buybacks/may-2020/coral-india-finance-and-housing-limited-letter-of-offer_46698.html
May 22, 2020    Jay Ushin Limited   https://www.sebi.gov.in/filings/takeovers/may-2020/jay-ushin-limited_46697.html
May 22, 2020    Pennar Industries - Post Buyback Public Announcement    https://www.sebi.gov.in/filings/buybacks/may-2020/pennar-industries-post-buyback-public-announcement_46696.html
May 22, 2020    Axis Global Equity Alpha Fund of Fund.  https://www.sebi.gov.in/filings/mutual-funds/may-2020/axis-global-equity-alpha-fund-of-fund-_46695.html
May 21, 2020    Axis Global Disruption Fund of Fund https://www.sebi.gov.in/filings/mutual-funds/may-2020/axis-global-disruption-fund-of-fund_46694.html
May 18, 2020    Reliance Industries Limited https://www.sebi.gov.in/filings/rights-issues/may-2020/reliance-industries-limited_46675.html
May 14, 2020    Public Advertisement of Spencer's Retail Limited    https://www.sebi.gov.in/filings/rights-issues/may-2020/public-advertisement-of-spencer-s-retail-limited_46693.html
May 12, 2020    Spencer's Retail Limited    https://www.sebi.gov.in/filings/rights-issues/may-2020/spencer-s-retail-limited_46692.html
May 12, 2020    Sequent Scientific Limited  https://www.sebi.gov.in/filings/takeovers/may-2020/sequent-scientific-limited_46662.html
May 11, 2020    Arvind Fashions Limited https://www.sebi.gov.in/filings/rights-issues/may-2020/arvind-fashions-limited_46659.html
May 05, 2020    JK Paper Limited - Public Announcement  https://www.sebi.gov.in/filings/buybacks/may-2020/jk-paper-limited-public-announcement_46647.html
May 05, 2020    Aurionpro Solutions Limited - Post BuyBack Advertisement    https://www.sebi.gov.in/filings/buybacks/may-2020/aurionpro-solutions-limited-post-buyback-advertisement_46646.html
May 04, 2020    KSOLVES INDIA LIMITED   https://www.sebi.gov.in/filings/public-issues/may-2020/ksolves-india-limited_46644.html
May 04, 2020    SBI ETF Consumption https://www.sebi.gov.in/filings/mutual-funds/may-2020/sbi-etf-consumption_46639.html
Page 3...

... and so on.

Upvotes: 1

FluidLight
FluidLight

Reputation: 461

It seems the website is making a POST request to getnewslistinfo.jsp and getting back the new table content as html. You can open up your Network (Ctrl+Shift+E on Firefox) then navigate to the next page and see the request being made and its parameters.

You can mimick that POST request and change the appropriate parameters for the next page (from what I saw it should be nextValue and doDirect) using urllib2 (or preferably requests). After you get the content you can simply parse it using BeautifulSoup and extract the a tags the way you already did.

Also a tip to you: You should separate your code into functions that do different things such as getPage(pageNum) that given a page number returns the html content, getLinks(html) that given an html page it gets all the links from the table and returns them as a list. This way your code will be more readable and easier to debug and use.

Upvotes: 1

Related Questions