CODEWITHSUNDEEP
pythonpandasdatetime
The Oracle
The Oracle

Reputation: 488

How to remove rows from a dataframe if before a certain date

Iam trying to remove all rows from a dataframe where dates from 'date' column are before 1-11-2019

The dataframe is produced by scraping google news (title, date, link, publisher). Here is the full code:

from bs4 import BeautifulSoup
import requests
import html5lib
import pandas as pd
import datetime

headers = {'User-Agent': 'Mozilla/5.0'}

#URL Generator (scraping news for 'sega')

urlA= 'https://news.google.com/search?q='
urlB='sega'
urlC='&hl=en-US&gl=US&ceid=US%3Aen'
url=urlA+urlB+urlC


response=requests.get(url)
soup=BeautifulSoup(response.content,'html5lib')
print(soup)


T=[]
t=[]
L=[]
P=[]

#Collecting Data
for  x in soup.find_all(class_='ipQwMb ekueJc RD0gLb'):
    title=x.text
    T.append(title)
    print(title)


for r in soup.find_all(class_='SVJrMe'):
    z=r.find('time')
    if z is not None:
        for y in r.find_all('time'):
            time=y.get('datetime')
            time=str(time).partition('T')
            time=time[0]
            time = datetime.datetime.strptime(time, "%Y-%m-%d").date()
            print(time)
            t.append(time)

    else:
        x='Not Specified'
        t.append(x)

for z in soup.find_all(class_='VDXfz'):
    links=z.get('href')
    links =links[1::] #removing the dot (first character always a 
dot in links which is not required)
    urlx= 'https://news.google.com'
    links= urlx+links
    L.append(links)

for w in soup.find_all(class_='wEwyrc AVN2gc uQIVzc Sksgp'):
    publisher = w.text
    P.append(publisher)

#Checking length to see all is equal
print(len(T))
print(len(t))
print(len(P))
print(len(L))

df=pd.DataFrame({'Title':(T) , 'Date':(t), 'Publisher' : (P), 'Link': (L)})

print(df)

Here is the current output (first 12 rows only):

enter image description here

As you can see the dataframe includes dates from before month of November, what i would like to do is delete all those rows. I have already converted the dates column into a 'dateTIME' format (see code [for r in soup.find...time=datetime.datetime.strip....].

Please advise line of code to add to achieve the required function. Please let me know if any clarification required.

Upvotes: 0

Views: 3326

Answers (2)

JR87
JR87

Reputation: 95

There are a couple of options. You can write a test in your loop to see if t (lower case t seems to be your date) is before November. If so, don't even append the other items to their respective lists.

There's also a method with dataframs called drop: https://chrisalbon.com/python/data_wrangling/pandas_dropping_column_and_rows/

You could also use this. Personally, once you have the t variable, I'd test it. If it meets your criteria for adding to the list, then add the others. If not, move on.

Upvotes: 0

Suraj Motaparthy
Suraj Motaparthy

Reputation: 520

IIUC, what you are looking for is:

df = df[df['Date']>datetime.date(2019,1,11)]

Upvotes: 1

Related Questions