Can't get text from p tag for some reason - Selenium (Python)

Question

I'm trying to scrape a page with Selenium. Sample HTML looks like this (from View Page Source)

<div class="col s12 m12 l4 xl4 therapist_contact_list">
<p class="col s6 m6 l6 xl6 noPaddingLeft lprofile-address hide-on-large-only"><i class="fa fa-map-marker" aria-hidden="true"></i> Birmingham, Alabama 35294</p>
<p class="col s12 read_content_par hide-on-large-only noPadding">Online Video & phone session only- etc
 </p>
</div>

so my selenium code is

location = listing.find_element_by_xpath('.//div[2]/p[1]').text
description = listing.find_element_by_xpath('.///div[2]/p[2]').text.replace(",","")

It's from a for loop, so the xpaths are correct. Everything else that I need to scrape works, but these two return blank. I don't know why.

I don't know if this means anything, but the other tags aren't P tags and they're not in quotation marks.

like so

and here

edit: page is https://www.goodtherapy.org/therapists/al/birmingham

Answer 1

@KunduK is correct. You can use element.get_attribute("textContent")to get the inner text of your element.

When looking over the page, I was able to loop through each therapist and collect their information and put it into a class object.

Therapist Data Class

class TherapistData:
    name = ""
    title = ""
    skills = ""
    profileUrl = ""
    isVerified = False
    verifiedCredentialsText = ""
    readContentPar = ""
    contactProfileAddress = ""
    contactReadContentPar = ""

I created a method called get_number_of_therapists that returned the number of therapists that displayed on the page. Then, I scraped the data for each therapist and put it into my class object

def scroll_to_element(self, xpath : str):
        element = self.chrome_driver.find_element(By.XPATH, xpath)
        self.chrome_driver.execute_script("return arguments[0].scrollIntoView();", element)


def get_number_of_therapists(self):
        return self.chrome_driver.find_elements(By.XPATH, "//ul[@class='therapist-list']//li").__len__()


def get_therapist_record_by_index(self, index : int):
        # Scroll to our record
        xpath = "//ul[@class='therapist-list']//li[{0}]".format(index)
        self.scroll_to_element(xpath)
        
        # Scrape our data
        therapist = TherapistData()
        therapist.profileUrl = self.chrome_driver.find_element(By.XPATH, "({0}//a[contains(@href, 'profile')])[1]".format(xpath)).get_attribute("href")
        therapist.name = self.chrome_driver.find_element(By.XPATH, "{0}//div[contains(@class, 'therapist_middle_section')]//h2".format(xpath)).text
        therapist.title = self.chrome_driver.find_element(By.XPATH, "{0}//div[contains(@class, 'therapist_middle_section')]//h3".format(xpath)).text
        therapist.skills = self.chrome_driver.find_element(By.XPATH, "{0}//div[contains(@class, 'therapist_middle_section')]//h4".format(xpath)).text
        
        therapist.verifiedCredentialsText = self.chrome_driver.find_element(
            By.XPATH,"{0}//div[contains(@class, 'therapist_middle_section')]//p[contains(@class, 'verified-credentials')]".format(xpath)).get_attribute("textContent").strip()
        
        therapist.isVerified = True if therapist.verifiedCredentialsText.find("Verified") != -1 else False
        
        therapist.readContentPar = self.chrome_driver.find_element(
            By.XPATH,"{0}//div[contains(@class, 'therapist_middle_section')]//p[contains(@class, 'read_content_par')]".format(xpath)).get_attribute("textContent").strip()
        
        therapist.contactProfileAddress = self.chrome_driver.find_element(
            By.XPATH, "{0}//div[contains(@class, 'therapist_contact_list')]//p[contains(@class, 'profile-address')]".format(xpath)).get_attribute("textContent").strip()
        
        therapist.contactReadContentPar = self.chrome_driver.find_element(
            By.XPATH, "{0}//div[contains(@class, 'therapist_contact_list')]//p[contains(@class, 'read_content_par')]".format(xpath)).get_attribute("textContent").strip()
        
        return therapist

Answer 2

The element is you are looking after is not visible on the page. That's the reason you are unable to get the value using .text.

Instead Use element.get_attribute("textContent")

To handle dynamic page induce WebDriverWait() and wait for presence_of_all_elements_located() and the iterate.

listing=WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"div.therapist_contact_list")))
for ele in listing:
    print(ele.find_element_by_xpath("./p[1]").get_attribute("textContent"))
    print(ele.find_element_by_xpath("./p[2]").get_attribute("textContent"))

You need to import below libraries.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Console Output:

Birmingham, Alabama 35294
Online Video & phone session only- Change begins with the first step. As a tele-mental health therapist I seek to walk on this journey with you, collaborate and provide a space where change is nurtured and encouraged. There will be tough moments, but with my warm and compassionate personality, I will encourage you to take the next step. I will pro

 Birmingham, Alabama 35209
As a counselor, I believe meeting an individual where they are is a vital part of providing best practice. With over seven years of experience assisting individuals, families, child/adolescents, and teens with a wide range of socials/personal relationships, emotional/trauma, behavioral/crisis, substance abuse, mental health issues, and many other c

 Birmingham, Alabama 35209
Thank you for visiting Love Out Loud Counseling and Consulting Services. My name is Stephanie Lett. As a Licensed Independent Clinical Social Worker, I have dedicated nearly a decade to serving others in order to improve society. My profession is much more than a tool for me to earn a livelihood-it is my passion. With an extensive background in c

 Birmingham, Alabama 35209
The focus of my practice, is to assist adults, children, and families identify the root, of their concerns. I assess situations in your life which attribute to current emotions, behaviors, and thought patterns. With child and adolescent clients my focus is similar, with the addition of close communication with the family. Moreover, with child and a

 Birmingham, Alabama 35223
Counseling is much more than problem solving. It's a place where you can celebrate your strengths and build on them to help prepare for life's challenges, big and small. I believe that first and foremost, successful counseling is built on trust and connection. Many people who struggle are reluctant to reach out. If you are seeking help you&

 Mountain Brook, Alabama 35223
I enjoy helping people understand why they do what they do or feel how they feel. I believe everyone that walks thru my door wants to feel better and together we can make that happen. My first priority is to provide a safe/non-judgmental environment. I take time getting to know what experiences have brought a person to my office. I provide init

 Birmingham, Alabama 35223
Taking the first step towards a better tomorrow for yourself can be difficult. Through your counseling journey, I want to help you feel safe & empowered as you work through your current roadblocks, both emotional & behavioral. I currently provide counseling sessions to both individuals and couples. I have training in mood & behavioral disorders, ad

 Homewood, Alabama 35209
I am a Licensed Professional Counselor in the state of Alabama with over 10 years experience working as a counselor, mentor, and life coach. I am licensed in Delaware as an LMHC and Georgia as an LPC. I provide in-office counseling with clients in the Birmingham, AL area and telehealth services for clients in Alabama, Delaware, and Georgia.
I

 Hoover, Alabama 35226
My approach to therapy is an integrative approach to wellness focusing on the whole person; Mind, Body, and Spirit. I utilize a person-centered and strength-based approach to help clients become empowered, to help them identify their strengths and help them utilize these strengths to succeed in life.
I understand from personal experience that a

 Birmingham, Alabama 35216
Creating a place in which my clients feel safe is important to me. Video sessions are a great choice for many clients. Other choices I offer are "Walk and Talk" sessions and in-office sessions. Being in nature can be very helpful for some people, so "Walk and Talk" sessions are very popular. My office is set up to be a calming place

Answer 3

You can use BeautifulSoup and Requests to do this instead of selenium. This is because using BeautifulSoup and Requests would reduce the execution time by about 10-20 seconds. This is how you do it:

from bs4 import BeautifulSoup
import requests

r = requests.get('https://www.goodtherapy.org/therapists/al/birmingham').text

soup = BeautifulSoup(r,'html5lib')

p_tags1 = soup.find_all('p',class_ = "col s6 m6 l6 xl6 noPaddingLeft lprofile-address hide-on-large-only")
p_tags2 = soup.find_all('p', class_ = "col s12 read_content_par hide-on-large-only noPadding")

for p in p_tags1:
    print(p.text.strip())

print("\n")

for p in p_tags2:
    print(p.text.strip())

Output:

Birmingham, Alabama 35294
Birmingham, Alabama 35209
Birmingham, Alabama 35209
Birmingham, Alabama 35209
Birmingham, Alabama 35223
Mountain Brook, Alabama 35223
Birmingham, Alabama 35223
Homewood, Alabama 35209
Hoover, Alabama 35226
Birmingham, Alabama 35216

Online Video & phone session only- Change begins with the first step. As a tele-mental health therapist I seek to walk on this journey with you, collaborate and provide a space where change is nurtured and encouraged. There will be tough moments, but with my warm and compassionate personality, I will encourage you to take the next step. I will pro
As a counselor, I believe meeting an individual where they are is a vital part of providing best practice. With over seven years of experience assisting individuals, families, child/adolescents, and teens with a wide range of socials/personal relationships, emotional/trauma, behavioral/crisis, substance abuse, mental health issues, and many other c
Thank you for visiting Love Out Loud Counseling and Consulting Services. My name is Stephanie Lett. As a Licensed Independent Clinical Social Worker, I have dedicated nearly a decade to serving others in order to improve society. My profession is much more than a tool for me to earn a livelihood-it is my passion. With an extensive background in c
The focus of my practice, is to assist adults, children, and families identify the root, of their concerns. I assess situations in your life which attribute to current emotions, behaviors, and thought patterns. With child and adolescent clients my focus is similar, with the addition of close communication with the family. Moreover, with child and a
Counseling is much more than problem solving. It's a place where you can celebrate your strengths and build on them to help prepare for life's challenges, big and small. I believe that first and foremost, successful counseling is built on trust and connection. Many people who struggle are reluctant to reach out. If you are seeking help you&
I enjoy helping people understand why they do what they do or feel how they feel. I believe everyone that walks thru my door wants to feel better and together we can make that happen. My first priority is to provide a safe/non-judgmental environment. I take time getting to know what experiences have brought a person to my office. I provide init
Taking the first step towards a better tomorrow for yourself can be difficult. Through your counseling journey, I want to help you feel safe & empowered as you work through your current roadblocks, both emotional & behavioral. I currently provide counseling sessions to both individuals and couples. I have training in mood & behavioral disorders, ad
I am a Licensed Professional Counselor in the state of Alabama with over 10 years experience working as a counselor, mentor, and life coach. I am licensed in Delaware as an LMHC and Georgia as an LPC. I provide in-office counseling with clients in the Birmingham, AL area and telehealth services for clients in Alabama, Delaware, and Georgia.
I
My approach to therapy is an integrative approach to wellness focusing on the whole person; Mind, Body, and Spirit. I utilize a person-centered and strength-based approach to help clients become empowered, to help them identify their strengths and help them utilize these strengths to succeed in life.
I understand from personal experience that a
Creating a place in which my clients feel safe is important to me. Video sessions are a great choice for many clients. Other choices I offer are "Walk and Talk" sessions and in-office sessions. Being in nature can be very helpful for some people, so "Walk and Talk" sessions are very popular. My office is set up to be a calming place