CODEWITHSUNDEEP
pythonhtmlpython-2.7beautifulsoupimaplib
Nyxynyx
Nyxynyx

Reputation: 63647

Line Breaks Prevent BeautifulSoup from Extracting Data

My script grabs HTML code from an email inbox via imaplib, passes it through BeautifulSoup and tries to extract all the hrefs in there.

rv, data = M.SEARCH(None, '(FROM "[email protected]")')
if rv == 'OK':
    for num in data[0].split():
        typ, data = M.fetch(num, '(RFC822)')
        html = data[0][1]

        soup = BeautifulSoup(html, 'lxml')
        for a in soup.find_all('a', href=True):
            print a['href']

However the html variable contains HTML code that have a new line every N characters, preventing BeautifulSoup from accurately returning the href, especially long ones that have been split up by the new line.

Theres also strange characters like =0D and 3D everywhere.

messages, <a=0D
href=3D"http://links.google.com/wf/click?upn=3DOGGGYNMPA980E3DmngbHusD=
Uo-2BK17XLM3ogFJfQXXXfMWZLdsQSSVv33HbPoHPXGcH8tSf9ZFFU5i-2FrV4O6ISlpDCIVaN5=
83xr1CGoa5yxZimagE5JiSUAhbZH8P7WiNvf35BsXrCxmrmRLMGB-2BJAQ-3D-3D_IcMuwcQVVt=
a699aeVjRRVxwBCNHkXaWO-2FyIlAqZ7CPsryDB24UVYZbMIvGLJb13chayC-2FLeucv-2FTrko=
7LaiaWHkzy85DWXrK1olI1SEJZs-2BMCAWfoVfloGJivlLSH0GQk0XeVT0j383tZrsymuWLF0S2=
q5j3LR91e76dRXQe7p8t5CgrBe-2FqGk6bmURG9XCNw3dwpHnymaR-2FggHQx6GnbbueF7PVp2H=
-2BGoHUEkMOSXJ8FfSgQIiGICvxz1zcBJPw-2FRoE3YDl-2By8XETkXjVaNchNA1ZN8FDCD5VUf=
V9oUOnavAirXX-2FEw1THfSpV4VYDX">unsubscribe</a></td>=0D
                </tr>=0D
                <tr>=0D
                    <td height=3D"12"></td>=0D
                </tr>=0D

What can we do to solve this problem?

Upvotes: 2

Views: 831

Answers (1)

Padraic Cunningham
Padraic Cunningham

Reputation: 180441

You can use quopri to decode the Quoted-printable data:

Quoted-Printable, or QP encoding, is an encoding using printable ASCII characters (alphanumeric and the equals sign "=") to transmit 8-bit data over a 7-bit data path or, generally, over a medium which is not 8-bit clean.1 It is defined as a MIME content transfer encoding for use in e-mail.

QP works by using the equals sign "=" as an escape character. It also limits line length to 76, as some software has limits on line length.

html = """<a=0D
href=3D"http://links.google.com/wf/click?upn=3DOGGGYNMPA980E3DmngbHusD=
Uo-2BK17XLM3ogFJfQXXXfMWZLdsQSSVv33HbPoHPXGcH8tSf9ZFFU5i-2FrV4O6ISlpDCIVaN5=
83xr1CGoa5yxZimagE5JiSUAhbZH8P7WiNvf35BsXrCxmrmRLMGB-2BJAQ-3D-3D_IcMuwcQVVt=
a699aeVjRRVxwBCNHkXaWO-2FyIlAqZ7CPsryDB24UVYZbMIvGLJb13chayC-2FLeucv-2FTrko=
7LaiaWHkzy85DWXrK1olI1SEJZs-2BMCAWfoVfloGJivlLSH0GQk0XeVT0j383tZrsymuWLF0S2=
q5j3LR91e76dRXQe7p8t5CgrBe-2FqGk6bmURG9XCNw3dwpHnymaR-2FggHQx6GnbbueF7PVp2H=
-2BGoHUEkMOSXJ8FfSgQIiGICvxz1zcBJPw-2FRoE3YDl-2By8XETkXjVaNchNA1ZN8FDCD5VUf=
V9oUOnavAirXX-2FEw1THfSpV4VYDX">unsubscribe</a></td>=0D
                </tr>=0D
                <tr>=0D
                    <td height=3D"12"></td>=0D
                </tr>=0D"""


from bs4 import BeautifulSoup
import quopri

soup = BeautifulSoup(quopri.decodestring(html), "lxml")
print(soup)
print(soup.select_one("a")["href"])

Will output:

<html><body><a href="http://links.google.com/wf/click?upn=OGGGYNMPA980E3DmngbHusDUo-2BK17XLM3ogFJfQXXXfMWZLdsQSSVv33HbPoHPXGcH8tSf9ZFFU5i-2FrV4O6ISlpDCIVaN583xr1CGoa5yxZimagE5JiSUAhbZH8P7WiNvf35BsXrCxmrmRLMGB-2BJAQ-3D-3D_IcMuwcQVVta699aeVjRRVxwBCNHkXaWO-2FyIlAqZ7CPsryDB24UVYZbMIvGLJb13chayC-2FLeucv-2FTrko7LaiaWHkzy85DWXrK1olI1SEJZs-2BMCAWfoVfloGJivlLSH0GQk0XeVT0j383tZrsymuWLF0S2q5j3LR91e76dRXQe7p8t5CgrBe-2FqGk6bmURG9XCNw3dwpHnymaR-2FggHQx6GnbbueF7PVp2H-2BGoHUEkMOSXJ8FfSgQIiGICvxz1zcBJPw-2FRoE3YDl-2By8XETkXjVaNchNA1ZN8FDCD5VUfV9oUOnavAirXX-2FEw1THfSpV4VYDX">unsubscribe</a>
<tr>
<td height="12"></td>
</tr> </body></html>
http://links.google.com/wf/click?upn=OGGGYNMPA980E3DmngbHusDUo-2BK17XLM3ogFJfQXXXfMWZLdsQSSVv33HbPoHPXGcH8tSf9ZFFU5i-2FrV4O6ISlpDCIVaN583xr1CGoa5yxZimagE5JiSUAhbZH8P7WiNvf35BsXrCxmrmRLMGB-2BJAQ-3D-3D_IcMuwcQVVta699aeVjRRVxwBCNHkXaWO-2FyIlAqZ7CPsryDB24UVYZbMIvGLJb13chayC-2FLeucv-2FTrko7LaiaWHkzy85DWXrK1olI1SEJZs-2BMCAWfoVfloGJivlLSH0GQk0XeVT0j383tZrsymuWLF0S2q5j3LR91e76dRXQe7p8t5CgrBe-2FqGk6bmURG9XCNw3dwpHnymaR-2FggHQx6GnbbueF7PVp2H-2BGoHUEkMOSXJ8FfSgQIiGICvxz1zcBJPw-2FRoE3YDl-2By8XETkXjVaNchNA1ZN8FDCD5VUfV9oUOnavAirXX-2FEw1THfSpV4VYDX

If you print the hex chars 3D and 0D, you can see it all makes sense:

In [4]: print("\x3D")
=

In [5]: print("\x0D")


In [6]:

Upvotes: 2

Related Questions