CODEWITHSUNDEEP
pythonseleniumselenium-webdriverweb-scrapingbeautifulsoup
Sultan Morbiwala
Sultan Morbiwala

Reputation: 171

Unable to access webpage with request in python

After some discussion with my problem on Unable to print links using beautifulsoup while automating through selenium

I realized that the main problem is in the URL which the request is not able to extract. URL of the page is actually https://society6.com/discover but I am using selenium to log into my account so the URL becomes https://society6.com/society?show=2

However, I can't use the second URL with request since its showing error. How do i scrap information from URL like this.

Upvotes: 1

Views: 218

Answers (1)

Moshe Slavin
Moshe Slavin

Reputation: 5204

You need to log in first!

To do that you can use the bs4.BeautifulSoup library.

Here is an implementation that I have used:

import requests
from bs4 import BeautifulSoup

BASE_URL = "https://society6.com/"


def log_in_and_get_session():
    """
    Get the session object with login details
    :return: requests.Session
    """    
    ss = requests.Session()
    ss.verify = False    # optinal for uncertifaied sites. 
    text = ss.get(f"{BASE_URL}login").text
    csrf_token = BeautifulSoup(text, "html.parser").input["value"]
    data = {"username": "your_username", "password": "your_password", "csrfmiddlewaretoken": csrf_token}
    # results = ss.post("{}login".format(BASE_URL), data=data)
    results = ss.post("{}login".format(BASE_URL), data=data)
    if results.ok:
        print("Login success", results.status_code)
        return ss
    else:
        print("Can't  login", results.status_code)

Using the 'post` method to log in...

Hope this helps you!

Edit

Added the beginning of the function.

Upvotes: 1

Related Questions