CODEWITHSUNDEEP
pythonpandasbeautifulsoup
JhonSnow
JhonSnow

Reputation: 27

Get the date of the scrape - Python & BeautifulSoup4

I want to report the exact date of the scrape in my CSV file using the following code:

import bs4
from bs4 import BeautifulSoup
import requests
import pandas
from pandas import DataFrame
import csv

with open('sortlist_scrap.csv', mode='w') as csv_file:
   fieldnames = ['Link', 'Rank', 'Date']
   writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
   writer.writeheader()

Found_link = []
Found_Rank = []
Found_date = []

url = "https://www.sortlist.fr/identite-visuelle/paris-fr?page={}"

for page_num in range(1, 100):
   page = requests.get(url.format(page_num))

link = soup.select_one('a[title*="Dupont Lewis"]')
previous_md_headlines = link.find_all_previous("a", {"class": "md-headline"})
next_md_headlines = link.find_all_next("a", {"class": "md-headline"})

position = 50 - len(next_md_headlines )
today = date.today()
page = sortlistscrap


Found_link.append(page)
Found_Rank.append(position)
Found_date.append(today)

data = { 'Found_Rank': Rank,
'Found_link':Found_link, 'Found_date':article_date}

df = DataFrame(data, columns = ['Found_link','Found_Rank','Found_date'])
df.to_csv(r'C:\Users\MacOS\Desktop\sortlist_scrap.csv')

But I get the following error: NameError: name 'date' is not defined

Can anyone help? Thanks!

Upvotes: 0

Views: 29

Answers (1)

wasif
wasif

Reputation: 15478

You might want to use datetime.datetime.today(). So first add from datetime import datetime to the start of code. And use today = datetime.today() instead of today = date.today()

Upvotes: 1

Related Questions