CODEWITHSUNDEEP
pythonweb-scraping
Farbod whatever
Farbod whatever

Reputation: 35

I can't web scrape scrape tr tags of a table using python

I am trying to scrape the proxy list of this site : http://cool-proxy.net/ However i can't find the tr tags with the class of "proxy-row ng-scope"

here is my code:

import requests
from bs4 import BeautifulSoup
url = 'http://cool-proxy.net/'
r = requests.get(url)
soup = BeautifulSoup(r.text , 'html.parser')
table = soup.find('table')
rows =table.find_all('tr', class_='proxy-row ng-scope')
print(rows)

Upvotes: 1

Views: 248

Answers (1)

Maryna Grushevskaya
Maryna Grushevskaya

Reputation: 51

try to get just, api url with all of the json, like:

import requests

response = requests.get('http://cool-proxy.net/proxies.json').json()

all_ips = [{pr['country_name']: pr['ip']} for pr in response]

print(all_ips)

I get only country_name, and ip, but there're all of the other keys what you want, like:

import requests

response = requests.get('http://cool-proxy.net/proxies.json').json()

all_ips = [{'country': pr['country_name'], 'ip': pr['ip'], 'port': pr['port']} for pr in response]

print(all_ips)

Upvotes: 1

Related Questions