CODEWITHSUNDEEP
pythonstringpython-3.xhttprequesttext-parsing
Arthur Codova
Arthur Codova

Reputation: 49

Python 3 - Getting some strings from a HTTPrequest response

I'm having a hard time extracting data from a httprequest response.

Can somebody help me? Here's a part of my code:

import requests

r = requests.get('https://www.example.com', verify=True)
keyword = r.text.find('loginfield')
print (keyword)

>>> 42136

42136 value basically means that string 'loginfield' exists on the response.text. But how do I extract specific strings from it?

Like for example I want to extract these exact strings:

<title>Some title here</title>

or this one:

<div id='bla...' #continues extracting of strings until it stops where I want it to stop extracting.

Anybody got an idea on how should I approach this problem?

Upvotes: 3

Views: 10428

Answers (2)

Sylvain Leroux
Sylvain Leroux

Reputation: 52040

Please note that for HTML-page data extraction, you should take a look at a specialized library like Beautiful soup. Your program will be less fragile and more maintainable that way.

string.find will return -1 if the string does not exists.

There is no string "loginfield" in the page you retrieved.

Once you have the correct index for your string, the returned value is the position of the first char of that string.

since you edited your question:

>>> r.text.find('loginfield') 
42136

That means, the string "loginfield" starts at offset 42136 in the text. You could display say 200 chars starting at that position that way:

>>> print(r.text[42136:42136+200])

To find the various values you looking for, you have to figure out where there are relative to that position.

Upvotes: 0

TerryA
TerryA

Reputation: 60024

You can use BeautifulSoup to parse HTML and get tags. Here's an example piece of code:

import requests
from bs4 import BeautifulSoup as BS
r = requests.get('https://www.example.com', verify=True)
soup = BS(r.text)
print(soup.find('title').text)

Should print:

Some title here

But depends on if it's the first title or not

Upvotes: 4

Related Questions