CODEWITHSUNDEEP
pythonpython-3.xweb-scrapingbeautifulsouppython-requests
MITHU
MITHU

Reputation: 154

Can't scrape information going back to the number of days I specify

I want to grab the posting date and title from this webpage using a python script that I've already created. The script is working errorlessly.

Now I wish to add some logic to the script so that it will only fetch stuff up to the date equivalent to the days I enter into days_to_go_back. For example, if I enter 2 right next to days_to_go_back, the script should scrape content up to October 6.

import requests
from bs4 import BeautifulSoup

link = 'https://atlanta.craigslist.org/search/lab'

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
}

days_to_go_back = 2

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link)
    soup = BeautifulSoup(res.text,"lxml")
    for item in soup.select("[id^='search-results'] li[class*='result']"):
        date = ' '.join(item.select_one("time.result-date").text.split())
        title = item.select_one("h3.result-heading > a").get_text(strip=True)
        print(date,title)

Current output:

Oct 7 General labor crew member needed
Oct 7 General Labor Demo
Oct 7 LAWN CARE & AERATION SPECIALISTS (ROSWELL, GA)
Oct 7 LAWN CARE & AERATION SPECIALISTS (ROSWELL, GA)
Oct 7 Craftsman Wanted
Oct 7 Home Cleaners Wanted! Make over $145+ for 1bd, $165 for 2bd cleans.
Oct 7 Sign Company In Search of Installers and Fabricators
Oct 7 ***PILE DRIVERS-DRILLERS-TRAINEES***
Oct 7 Warehouse Package Handler
Oct 7 Workers Needed for Local Fair (Cash Paid Daily)
Oct 7 Warehouse Team Member - Multiple Shifts Available
Oct 7 Construction
Oct 7 Lyft Driver (part-time / full-time)
Oct 7 Lyft Driver (part-time / full-time)
Oct 7 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 7 Warehouse Team Member - $3,000 Sign-On Bonus
Oct 7 Warehouse Team Member - $3,000 Sign-On Bonus
Oct 7 Amazon Warehouse Assistant - Immediate Hire
Oct 7 Amazon Picker - Overnight Shifts Available
Oct 7 Warehouse Team Member - With Sign-On Bonus!
Oct 7 Amazon Warehouse Assistant
Oct 7 Need a Crew of 4+
Oct 6 Helper and Construction
Oct 6 Truck Driver/El Camionero
Oct 6 Workers Needed for Local Fair (Cash Paid Daily)
Oct 6 Roofing Crews Needed! ($2,000 Sign On Bonus - Start Today!)
Oct 6 Available Immediately General Labor/Driver with Truck or Van Positions
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 Automotive Technicians Needed
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 Automotive Technicians Needed
Oct 6 Automotive Technicians Needed
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 6 $20-25/hr. SIMPLE VALET TRASH COLLECTOR-Camp Creek/East Point 30331
Oct 6 Asphalt workers labors
Oct 6 CDL-A Company Drivers - $5K SIGN-ON BONUS! 💰 🚚
Oct 6 EXPERIENCED HANDYMAN ***2 immediate openings
Oct 6 Automotive Detail /Porter/Shop Help
Oct 6 Fork Lift Operator/ Order Picker
Oct 6 Lot Operations
Oct 6 Waterproofing Foreman
Oct 6 Auto Detailer
Oct 6 Lyft Driver (part-time / full-time)
Oct 6 Lyft Driver (part-time / full-time)
Oct 6 FULL TIME WITH PAID BENEFITS $16+ HOUR
Oct 5 Georgia Canvasser
Oct 5 💚 CLEANING JOBS AVAILABLE 💚  $22+/hr, Paid Daily 💚
Oct 5 Farm Help Animal Care
Oct 5 Warehouse Team Member - $3,000 Sign-On Bonus
Oct 5 Maintenance/Custodial
Oct 5 Hiring House Keeper For Airbnb
Oct 5 Warehouse Material Handler: $15.92/hr,M-F/Full Time/Day Shift/Benefits
Oct 5 Laborers, Forman, and Drivers
Oct 5 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 5 CANVASS POSITION (MAKE MONEY THIS WEEK)
Oct 5 Hiring full Auto Detailer
Oct 5 Amazon Packer - Part-time Night Shifts Available
Oct 5 Warehouse Team Member Sun - Wed (06:30PM - 05:00AM)
Oct 5 Amazon Warehouse Assistant - Immediate Hire
Oct 5 EC3 Construction Is Now Hiring Framers
Oct 5 Detailer Wanted $16.00hr. plus 250.00 BONUS$$$
Oct 5 FREE $10 CASH FOR EVERYONE GET IT TODAY! SUPPLIES LIMITED HURRY
Oct 5 Now Hiring: Diesel technicians
Oct 5 Warehouse Associate
Oct 5 Delivery Driver Needed
Oct 5 Drivers Helper - Weekly Pay
Oct 5 Landscape Crew Member
Oct 5 Seasonal Laborers
Oct 5 Seasonal Laborers
Oct 5 WINDOW AND DOOR INSTALLERS NEEDED
Oct 5 Commercial Roofers and Helpers
Oct 5 Lyft Driver (part-time / full-time)
Oct 5 Lyft Driver (part-time / full-time)
Oct 5 Warehouse Associate/Driver
Oct 5 Grand Opening! Now Hiring! Immediate work!
Oct 5 C.D.L Drivers wanted
Oct 5 SET UP Workers Needed for Local Fair (Cash Paid Daily)
Oct 5 Workers Needed for Local Fair (Cash Paid Daily)
Oct 4 Distribution Center General Worker Position
Oct 4 Distribution Center General Worker Position
Oct 4 Distribution Center General Worker Position
Oct 4 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 4 Warehouse Worker
Oct 4 House Cleaner Needed
Oct 4 Janitor Job offer
Oct 4 HIRING TIRE TECHNICIANS
Oct 4 CDL-A Company Drivers - $5K SIGN-ON BONUS! 💰 🚚
Oct 4 Part Time Groundskeeper/Cleaner
Oct 4 Driver and helper
Oct 4 Full-Time Community Trash Collector
Oct 4 * Deliver with DoorDash *
Oct 4 Experienced maintenance men
Oct 4 Maintenance Manager for Luxury Apts!
Oct 4 Maintenance Manager for Luxury Apts!
Oct 4 Maintenance Tech II for Luxury Apts!
Oct 4 Maintenance Tech II for Luxury Apts!
Oct 4 Maintenance Tech I for Luxury Apts!
Oct 4 Stucco Repair Help
Oct 4 Lyft Driver (part-time / full-time)
Oct 4 Lyft Driver (part-time / full-time)
Oct 4 Workers Needed for Local Fair (Cash Paid Daily)
Oct 4 Warehouse Team Member - Multiple Shifts Available
Oct 4 🌊 Tidal Wave Lawrenceville ǀ Car Wash Attendant 🌊
Oct 4 🌊 Tidal Wave Kennesaw ǀ Car Wash Attendant 🌊
Oct 4 🌊 Tidal Wave Gainesville ǀ Car Wash Attendant 🌊
Oct 4 🌊 Tidal Wave Dacula ǀ Car Wash Attendant 🌊
Oct 4 🌊 Tidal Wave Canton ǀ Car Wash Attendant 🌊
Oct 4 Heavy equipment operator
Oct 4 **Full Time Building Cleaner (pressure washing/brush scrubbing) Needed**
Oct 4 Amazon Picker - Up to $16.5/hr
Oct 4 Warehouse Laborer - Immediate Hire
Oct 4 Warehouse Team Member - Split Shifts (20hr/week)
Oct 4 Warehouse Team Member - Morning Shifts Available
Oct 4 Warehouse Team Member - Multiple Shifts Available

Expected output (up to oct 6):

Oct 7 General labor crew member needed
Oct 7 General Labor Demo
Oct 7 LAWN CARE & AERATION SPECIALISTS (ROSWELL, GA)
Oct 7 LAWN CARE & AERATION SPECIALISTS (ROSWELL, GA)
Oct 7 Craftsman Wanted
Oct 7 Home Cleaners Wanted! Make over $145+ for 1bd, $165 for 2bd cleans.
Oct 7 Sign Company In Search of Installers and Fabricators
Oct 7 ***PILE DRIVERS-DRILLERS-TRAINEES***
Oct 7 Warehouse Package Handler
Oct 7 Workers Needed for Local Fair (Cash Paid Daily)
Oct 7 Warehouse Team Member - Multiple Shifts Available
Oct 7 Construction
Oct 7 Lyft Driver (part-time / full-time)
Oct 7 Lyft Driver (part-time / full-time)
Oct 7 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 7 Warehouse Team Member - $3,000 Sign-On Bonus
Oct 7 Warehouse Team Member - $3,000 Sign-On Bonus
Oct 7 Amazon Warehouse Assistant - Immediate Hire
Oct 7 Amazon Picker - Overnight Shifts Available
Oct 7 Warehouse Team Member - With Sign-On Bonus!
Oct 7 Amazon Warehouse Assistant
Oct 7 Need a Crew of 4+
Oct 6 Helper and Construction
Oct 6 Truck Driver/El Camionero
Oct 6 Workers Needed for Local Fair (Cash Paid Daily)
Oct 6 Roofing Crews Needed! ($2,000 Sign On Bonus - Start Today!)
Oct 6 Available Immediately General Labor/Driver with Truck or Van Positions
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 Automotive Technicians Needed
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 Automotive Technicians Needed
Oct 6 Automotive Technicians Needed
Oct 6 AUTOMOTIVE / TIRE TECHNICIAN ($800 New Hire Bonus)
Oct 6 NEED HANDYMAN FOR TASKS - at least $35/hr, Paid Daily
Oct 6 $20-25/hr. SIMPLE VALET TRASH COLLECTOR-Camp Creek/East Point 30331
Oct 6 Asphalt workers labors
Oct 6 CDL-A Company Drivers - $5K SIGN-ON BONUS! 💰 🚚
Oct 6 EXPERIENCED HANDYMAN ***2 immediate openings
Oct 6 Automotive Detail /Porter/Shop Help
Oct 6 Fork Lift Operator/ Order Picker
Oct 6 Lot Operations
Oct 6 Waterproofing Foreman
Oct 6 Auto Detailer
Oct 6 Lyft Driver (part-time / full-time)
Oct 6 Lyft Driver (part-time / full-time)
Oct 6 FULL TIME WITH PAID BENEFITS $16+ HOUR

Upvotes: 0

Views: 52

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195418

Try:

import requests
from datetime import datetime
from bs4 import BeautifulSoup

link = "https://atlanta.craigslist.org/search/lab"

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36",
}

days_to_go_back = 2

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link)
    soup = BeautifulSoup(res.text, "lxml")

    now = datetime.now()

    for item in soup.select("[id^='search-results'] li[class*='result']"):
        date = " ".join(item.select_one("time.result-date").text.split())
        title = item.select_one("h3.result-heading > a").get_text(strip=True)

        d = datetime.strptime(item.time["datetime"][:10], "%Y-%m-%d")
        if (now - d).days < days_to_go_back:
            print(date, title)

Prints:


...
Oct 6 Automotive Detail /Porter/Shop Help
Oct 6 Fork Lift Operator/ Order Picker
Oct 6 Lot Operations
Oct 6 Waterproofing Foreman
Oct 6 Auto Detailer
Oct 6 Lyft Driver (part-time / full-time)
Oct 6 Lyft Driver (part-time / full-time)
Oct 6 FULL TIME WITH PAID BENEFITS $16+ HOUR

Upvotes: 2

Related Questions