CODEWITHSUNDEEP
pythonhtmlbeautifulsouphtml-parsing
Supriya Bhart
Supriya Bhart

Reputation: 89

How to scrape data from a website and store in folders in a hierarchical stucture using python script?

I am trying to fetch data from a website using python, I have locally downloaded the text file then, I want to fetch the data from an anchor tag inside the text file and make a new folder renamed as the anchor tag value.

Here I am making new folders using python but unable to fetch the data from the txt file and process

I am able to make new folders but i want the rename as the anchor tag value

import os

root_path = '/home'    
folders = ['folder 01', 'folder 02', 'folder 03']    
for folder in folders:
    os.mkdir(os.path.join(root_path, folder))

Upvotes: 0

Views: 386

Answers (1)

KunduK
KunduK

Reputation: 33384

Please try the below code.It should create all folders name present in anchor tag.

from bs4 import BeautifulSoup
import requests
import os
url = 'Url Here'
html = requests.get(url).text
soup = BeautifulSoup(html, 'html.parser')
for folder in soup.find_all('a'):
    root_path = '/home'
    os.mkdir(os.path.join(root_path, folder.text))

Please let me know if this work.

Upvotes: 1

Related Questions