Python BeautifulSoup and requests webscraping

Question

I am trying to get the generated phrase from the following website as a string. https://randomwordgenerator.com/phrase.php I've looked through the html and I believe I've determined where the phrase is located in the html structure.

This is a bit of the html nearby.

<div id="loading_result" class="small-img-results">
    <ol id="result">
        <li> == $0
            <div>
                <span class="support-phrase">Generated Phrase </span>
                <span class="subtle">...</span>
            </div>
        </li>
    </ol>
</div>

In this case, I want the text "Generated Phrase"

This is what I am currently doing

pageLink = "https://randomwordgenerator.com/phrase.php"
pageResponse = requests.get(pageLink, timeout=5)
pageContent = BeautifulSoup(pageResponse.content, "html.parser")

span = pageContent.find_all("span", {"class": "support-phrase"})

The issue is the value of span after this runs is an empty list. I'm new to beautiful soup so this might be a really simple problem, but I have not found anything particularly clear that solves this problem.

Thanks in advance!

edit: I'm now wondering if the issue is that this particular span I'm looking for is nested inside of an which is in a series of divs in the body.

Answer 1

Random Phrase of this page loads through XHR. So you can not find phrase using BeautifulSoup. Use XHR request to get list of phrase.

import requests

pageLink = "https://randomwordgenerator.com/json/phrases.json"

pageResponse = requests.get(pageLink, timeout=5)
print pageResponse.status_code, pageResponse.content

output:

{"data":[ {"phrase":"A Chip on Your Shoulder","meaning":"Being angry about something that happened in the past."}, {"phrase":"A Dime a Dozen","meaning":"Something that is extremely common."}, {"phrase":"A Fool and His Money are Soon Parted","meaning":"It's easy for a fool to lose his/her money."}, ... ]}

Answer 2

You will need selenium to get the exact value(s) shown on the page. The reason for this is that whilst the total phrases (134) are returned in an array from an xhr (https://randomwordgenerator.com/json/phrases.json) which returns json; the actual index/indices (e.g. function randomiseUniqueNumbers) to select from this array, the order of items in the array (e.g. Array.prototype.shuffle = function() ) , and the rules which handle potential clashes I think (e.g. function getResults) are all defined in a js file https://randomwordgenerator.com/assets/js-compress/f0351bd03da6dab13a24355fa7deeabd.js?v=1577899960:formatted. The first two of those, at least, use random number generation between the bounds of the array size. There is no seeding and, whilst I think you can write your own versions of these, you are not guaranteed to get the same result as on the page - in fact you are more likely to get a different phrase.

Outline selenium

from selenium import webdriver

d = webdriver.Chrome()
d.get('https://randomwordgenerator.com/phrase.php')
print([i.text for i in d.find_elements_by_css_selector('.support-phrase')])

For a single phrase just use

d.find_element_by_css_selector('.support-phrase').text

Answer 3

If you Go To NetWork Tab under XHR you will get following json url which returns phrases and subtle in json format.You can achieve that without selenium and Beautifulsoup

https://randomwordgenerator.com/json/phrases.json

---

Code Here:

import requests
url='https://randomwordgenerator.com/json/phrases.json'
response=requests.get(url).json()
print(response['data'])
#Print phrase
print(response['data'][0]['phrase'])
#Print meaning
print(response['data'][0]['meaning'])

---

Output here:

[{'meaning': 'Being angry about something that happened in the past.', 'phrase': 'A Chip on Your Shoulder'}, {'meaning': 'Something that is extremely common.', 'phrase': 'A Dime a Dozen'}, {'meaning': "It's easy for a fool to lose his/her money.", 'phrase': 'A Fool and His Money are Soon Parted'}, {'meaning': 'A task that is simple to accomplish.', 'phrase': 'A Piece of Cake'}, {'meaning': 'Something that is extremely expensive.', 'phrase': 'An Arm and a Leg'}, {'meaning': 'When something is incomprehensible due to complexity; unintelligble.', 'phrase': 'All Greek To Me'}, {'meaning': 'To go back to the beginning; back to the drawing board.', 'phrase': 'Back to Square One'}, {'meaning': 'Starting over again on a new design from a previously failed attempt.', 'phrase': 'Back To the Drawing Board'}, {'meaning': 'To make a wrong assumption about something.', 'phrase': 'Barking Up The Wrong Tree'}, {'meaning': 'To bring up an issue that has already been resolved.', 'phrase': 'Beating a Dead Horse'}, {'meaning': 'Someone who is beating around the bush is someone who avoids the main point.', 'phrase': 'Beating Around the Bush'}, {'meaning': 'Being faced with two difficult choices.', 'phrase': 'Between a Rock and a Hard Place'}, {'meaning': 'People tend to associate with others who share similar interests or values.', 'phrase': 'Birds of a Feather Flock Together'}, {'meaning': 'Breaking down a social stiffness.', 'phrase': 'Break The Ice'}, {'meaning': "To ruin someone's happy moment.", 'phrase': 'Burst Your Bubble'}, {'meaning': 'Coming close to a successful outcome only to fall short at the end.', 'phrase': 'Close But No Cigar'}, {'meaning': "It's useless to worry about things that\xa0 already happened and cannot be changed.", 'phrase': 'Cry Over Spilt Milk'}, {'meaning': 'Someone that calls for help when it is not needed. Someone who is lying.', 'phrase': 'Cry Wolf'}, {'meaning': 'A cup of joe is an American nickname for a cup of coffee.', 'phrase': 'Cup Of Joe'}, {'meaning': 'Typically said to indicate that any further investigation into a situation may lead to harm.', 'phrase': 'Curiosity Killed The Cat'}, {'meaning': 'To cut the mustard is to meet a required standard, or to meet expectations.', 'phrase': 'Cut The Mustard'}, {'meaning': 'To get to the point, leaving out all of the unnecessary details.', 'phrase': 'Cut To The Chase'}, {'meaning': 'Do not rely on something you are not sure of.', 'phrase': "Don't Count Your Chickens Before They Hatch"}, {'meaning': 'When you receive a gift from someone, do not be ungrateful.', 'phrase': "Don't Look a Gift Horse In The Mouth"}, {'meaning': '(1) A term used in a boxing. (2) Someone who has become incapacitated.', 'phrase': 'Down And Out'}, {'meaning': 'Someone or something that looks to be defeated, or nearly so.', 'phrase': 'Down For The Count'}, {'meaning': 'Practical or humble; unpretentious.', 'phrase': 'Down To Earth'}, {'meaning': 'A tense situation where the outcome is decided only in the last few seconds.', 'phrase': 'Down To The Wire'}, {'meaning': 'Failing to recall a memory. Unable to remember something.', 'phrase': 'Drawing a Blank'}, {'meaning': 'To greatly frustrate someone. To drive someone crazy, insane, bonkers, or bananas.', 'phrase': 'Drive Me Nuts'}, {'meaning': 'To fall down ill or to die in large numbers.', 'phrase': 'Dropping Like Flies'}, {'meaning': 'Something that is easy.', 'phrase': 'Easy As Pie'}, {'meaning': 'Having confidence in a specific outcome; being almost sure about something.', 'phrase': 'Eat My Hat'}, {'meaning': 'Ignoring a large, obvious problem or failing to address an issue that stands out in a major way.', 'phrase': 'Elephant in the Room'}, {'meaning': 'Something that is all over.', 'phrase': 'Elvis Has Left The Building'}, {'meaning': 'To be optimistic, even in difficullt times.', 'phrase': 'Every Cloud Has a Silver Lining'}, {'meaning': 'Including nearly everything possible.', 'phrase': 'Everything But The Kitchen Sink'}, {'meaning': 'To retaliate with an attack that is similar to the attack used against you.', 'phrase': 'Fight Fire With Fire'}, {'meaning': 'Someone being in a situation that they are unfamiliar or unsuited for.', 'phrase': 'Fish Out Of Water'}, {'meaning': 'Being fit as a fiddle means to be in perfect health.', 'phrase': 'Fit as a Fiddle'}, {'meaning': 'A type of bazaar where inexpensive goods are sold or bartered.', 'phrase': 'Flea Market'}, {'meaning': 'To be enraged and show it.', 'phrase': 'Foaming At The Mouth'}, {'meaning': 'Iron pyrities is a worthless mineral that resembles gold.', 'phrase': "Fool's Gold"}, {'meaning': "It's better to teach a person how to do something than to do that something for them.", 'phrase': 'Give a Man a Fish'}, {'meaning': 'To risk it all, even if it means losing everything. To go all out.', 'phrase': 'Go For Broke'}, {'meaning': 'Putting yourself in a risky situation in order to help someone; or to hazard a guess.', 'phrase': 'Go Out On a Limb'}, {'meaning': 'A smugly virtuous person.', 'phrase': 'Goody Two-Shoes'}, {'meaning': 'Very fast or quick.', 'phrase': 'Greased Lightning'}, {'meaning': "Anything that's easy or has no difficulty; something that is a certainty.", 'phrase': 'Hands Down'}, {'meaning': 'The state of being happy; feeling delighted.', 'phrase': 'Happy as a Clam'}, {'meaning': "Something that's difficult to accept.", 'phrase': 'Hard Pill to Swallow'}, {'meaning': 'Falling deeply in love with another person.', 'phrase': 'Head Over Heels'}, {'meaning': 'Used as an advanced warning. To become keenly aware.', 'phrase': 'Heads Up'}, {'meaning': 'A shout of agreement, or to draw attention to a speaker.', 'phrase': 'Hear, Hear'}, {'meaning': 'To be left behind; abandoned. Being in a helpless situation without a way to recover.', 'phrase': 'High And Dry'}, {'meaning': 'A boxing term. Also often used to refer to inappropriate words, or comments that are too personal.', 'phrase': 'Hit Below The Belt'}, {'meaning': 'A feeling that something is not quite right, or awry.', 'phrase': 'I Smell a Rat'}, {'meaning': 'One should discontinue with a task if they are unable to cope with it due to pressure.', 'phrase': "If You Can't Stand the Heat, Get Out of the Kitchen"}, {'meaning': 'Being in a difficult predicament; a mess; an undesirable situation.', 'phrase': 'In a Pickle'}, {'meaning': 'Losing money. Being in debt.', 'phrase': 'In the Red'}, {'meaning': 'Failing to meet expectations; not being as good as people say.', 'phrase': "It's Not All It's Cracked Up To Be"}, {'meaning': "A task that's easy to accomplish, a thing lacking complexity.", 'phrase': "It's Not Brain Surgery"}, {'meaning': 'Having suitable skill in multiple things, but not being an expert in any of them.', 'phrase': 'Jack of All Trades Master of None'}, {'meaning': 'Being in a dangerous or very deadly situation.', 'phrase': 'Jaws of Death'}, {'meaning': 'Usually this references a tool used by rescuers when they pry or cut open a car to save the occupant.', 'phrase': 'Jaws of Life'}, {'meaning': 'For a ruse or trick to be discovered; to be caught.', 'phrase': 'Jig Is Up'}, {'meaning': 'Something that occurs too early before preparations are ready. Starting too soon.', 'phrase': 'Jumping the Gun'}, {'meaning': 'To keep going, pressing forward; never stopping.', 'phrase': "Keep On Truckin'"}, {'meaning': 'To be watchful; paying careful attention to something.', 'phrase': 'Keep Your Eyes Peeled'}, {'meaning': 'Keeping calm. Usually said by someone who is trying to avoid making others upset.', 'phrase': 'Keep Your Shirt On'}, {'meaning': 'To be taken by surprise.', 'phrase': 'Knock Your Socks Off'}, {'meaning': 'Having a familiarity or understanding of how something works.', 'phrase': 'Know the Ropes'}, {'meaning': 'Getting sincere about something; applying oneself seriously to a job.', 'phrase': 'Knuckle Down'}, {'meaning': "Permission to start, or it could mean 'go faster!'", 'phrase': 'Let Her Rip'}, {'meaning': 'To go at a quick pace; no delaying!', 'phrase': 'Lickety Split'}, {'meaning': "Resembling one's parents in terms of appearance or behavior.", 'phrase': 'Like Father Like Son'}, {'meaning': 'Old in age. Mainly used when referring to people or horses.', 'phrase': 'Long In The Tooth'}, {'meaning': 'A pair of people who have a shared love for each other.', 'phrase': 'Love Birds'}, {'meaning': 'The affectionate stuff that people do when they are in love, such as kissing and hugging.', 'phrase': 'Lovey Dovey'}, {'meaning': 'A person who does not speak a great deal; someone who talks with as few words as possible.', 'phrase': 'Man of Few Words'}, {'meaning': "Suggests that money is a resource that must be earned and is not one that's easily acquired.", 'phrase': "Money Doesn't Grow On Trees"}, {'meaning': 'One who escalates small things and turns them into big problems.', 'phrase': 'Mountain Out of a Molehill'}, {'meaning': 'Delicious; something that looks or tastes appetizing.', 'phrase': 'Mouth-watering'}, {'meaning': 'Someone or something that one finds to be agreeable or delightful.', 'phrase': 'My Cup of Tea'}, {'meaning': 'Something that is impossible or extremely difficult to find, especially because the area you have to search is too large.', 'phrase': 'Needle In a Haystack'}, {'meaning': 'Anything that requires minimal brain activity to accomplish.', 'phrase': 'No-Brainer'}, {'meaning': 'Finishing a task without making any excuses.', 'phrase': 'No Ifs, Ands, or Buts'}, {'meaning': "Someone who isn't witty or sharp, but rather, they are ignorant, unintelligent, or senseless.", 'phrase': 'Not the Sharpest Tool in the Shed'}, {'meaning': 'A person that is crazy or behaving in idiotic ways', 'phrase': "Off One's Base"}, {'meaning': 'Having strong feelings of happiness or satisfaction.\xa0', 'phrase': 'On Cloud Nine'}, {'meaning': 'Being in a situation that looks to be hopeless!', 'phrase': 'On the Ropes'}, {'meaning': 'Thinking alike or understanding something in a similar way with others.', 'phrase': 'On the Same Page'}, {'meaning': 'What you would expect to happen; something normal or common.', 'phrase': 'Par For the Course'}, {'meaning': 'Said when things are about to get serious.', 'phrase': 'Playing For Keeps'}, {'meaning': 'Pretending to be dead, or to be deceitful about something.', 'phrase': 'Playing Possum'}, {'meaning': 'A situation that has gotten way more serious or interesting due to recent complexities or developments.', 'phrase': 'Plot Thickens\xa0- The'}, {'meaning': 'Making fun of something or someone; ridicule.', 'phrase': 'Poke Fun At'}, {'meaning': 'Asking someone to be quiet or to shut up.', 'phrase': 'Put a Sock In It'}, {'meaning': 'Spending time with another to strengthen the relationship.', 'phrase': 'Quality Time'}, {'meaning': "Things that are fixed with great speed, but as a result, it's probably not going to work very well.", 'phrase': 'Quick and Dirty'}, {'meaning': 'Performing an action with the greatest of haste.', 'phrase': 'Quick On the Draw'}, {'meaning': "To spoil someone's fun or plans; ruining a pleasurable moment", 'phrase': 'Rain on Your Parade'}, {'meaning': 'When it is raining heavily.', 'phrase': 'Raining Cats and Dogs'}, {'meaning': "Often said by the winner in poker, as the others 'weep' over the loss.", 'phrase': "Read 'Em and Weep"}, {'meaning': 'A cheer people yell, usually at rodeos when cowboys are clinging to the backs of untamed horses.', 'phrase': 'Ride Him, Cowboy!'}, {'meaning': 'Immediately, done in a hurry; without delay.', 'phrase': 'Right Off the Bat'}, {'meaning': 'Right from the beginning; to do something from the start.', 'phrase': 'Right Out of the Gate'}, {'meaning': 'Recalling a memory; causing a person to remember something or someone.', 'phrase': 'Ring Any Bells?'}, {'meaning': 'To tolerate or endure through the unexpected mishappenings you may encounter from time to time.', 'phrase': 'Roll With the Punches'}, {'meaning': 'Getting away freely from custody, punishment, or any type of risky situation.', 'phrase': 'Scot-free'}, {'meaning': 'Getting the bad end of a deal, or receiving the least desirable outcome from something.', 'phrase': 'Short End of the Stick'}, {'meaning': 'An attempt that has little chance for success.', 'phrase': 'Shot In the Dark'}, {'meaning': 'A person, usually one who is behaving badly.', 'phrase': 'Son of a Gun'}, {'meaning': 'Giving something your all.', 'phrase': 'Swinging For the Fences'}, {'meaning': 'Supporting what you say, not just with words, but also through action or evidence.', 'phrase': 'Talk the Talk'}, {'meaning': 'To not work alone, but rather, together with others in order to achieve a certain goal.', 'phrase': "There's No I in Team"}, {'meaning': 'Giving up; to surrender.', 'phrase': 'Throw In the Towel'}, {'meaning': "High quality, exceptional; something that's very valuable.", 'phrase': 'Top Drawer'}, {'meaning': 'To remain resillient even in hard times; enduring.', 'phrase': 'Tough It Out'}, {'meaning': 'It can refer to the popular rope pulling game or it can mean a struggle for authority.', 'phrase': 'Tug of War'}, {'meaning': 'Two things have been completed, but there is one more that has yet to be finished.', 'phrase': 'Two Down, One to Go'}, {'meaning': 'One who may seem plain at first in appearance or capability, but later turns out to be beautiful or great.', 'phrase': 'Ugly Duckling'}, {'meaning': 'Not feeling well, in health or mood.', 'phrase': 'Under the Weather'}, {'meaning': 'Missing something that should be really obvious.', 'phrase': 'Under Your Nose'}, {'meaning': 'Angry; being roused to the point that you are ready to fight.', 'phrase': 'Up In Arms'}, {'meaning': "An occurance of sorts that brings a problem to somebody's attention and they realize it needs fixing.", 'phrase': 'Wake Up Call'}, {'meaning': 'Things that go up must eventually return to the earth due to gravity.', 'phrase': 'What Goes Up Must Come Down'}, {'meaning': 'A rhetorical question used by a person who feels they are being given less consideration than someone else.', 'phrase': 'What Am I, Chopped Liver?'}, {'meaning': 'When something is about to begin, get serious, or put to the test.', 'phrase': 'When the Rubber Hits the Road'}, {'meaning': 'Futilely pursuing something that will never be attainable.', 'phrase': 'Wild Goose Chase'}, {'meaning': 'Nonviolent; someone who is mild or gentle.', 'phrase': "Wouldn't Harm a Fly"}, {'meaning': "A way to notify a person that what they're saying is predictable or boring.", 'phrase': 'Yada Yada'}, {'meaning': "Don't judge someone or something only by the outward appearance.", 'phrase': "You Can't Judge a Book By Its Cover"}, {'meaning': 'It can be challenging to teach a person something new.', 'phrase': "You Can't Teach an Old Dog New Tricks"}]

phrase

A Chip on Your Shoulder

meaning

Being angry about something that happened in the past.

Answer 4

from lxml import html
import requests

page = requests.get('https://randomwordgenerator.com/phrase.php')
# get xpath of the string and at end '/text()'
text_xpath= '/html/body/div[2]/div/div[3]/div/div/div[1]/h1/text()' 
tree = html.fromstring(page.content)
page_text =tree.xpath(text_xpath)
print(page_text)