CODEWITHSUNDEEP
pythonxpathscrapy
ndewan
ndewan

Reputation: 21

how to use a variable which has tag info in xpath to extract

I am writing a scrapy code, which takes the url, the tags where my data to be scraped is stored from a csv file. That tag I am assigning to a temporary variable like z(having h1), av(having title), an(having td) etc. When I am using that variable in xpath , it does not extract anything out of that tag. Can anyone help me ?

import pandas as pd
import requests
from scrapy.http import TextResponse

x=pd.read_csv(r'C:\Users\ndewan\Desktop\pattern-swage.csv', encoding="utf-8")
x=x.replace('<','',regex=True).replace('>','',regex=True)

url=str(x.iloc[0,0]) 
r = requests.get(url)
response = TextResponse(r.url, body=r.text, encoding='utf-8')
z=x.iloc[0,1]
s=x.iloc[0,2]
an=x.iloc[0,3]
av=x.iloc[0,4]
part=response.xpath('//av//text()').extract()
print(part)
print(url)
print(z)

Upvotes: 1

Views: 199

Answers (1)

Reid Ballard
Reid Ballard

Reputation: 1525

Since you're using variables, you should denote them with $ and supply their value as a keyword argument per the docs.

part=response.xpath('//$av//text()', av = av).extract()

Alternatively, if you prefer, use string formatting.

part=response.xpath(f'//{av}//text()').extract()

Upvotes: 1

Related Questions