CODEWITHSUNDEEP
pythonxpathscrapy
PyRar
PyRar

Reputation: 549

Not retrieving data from script tag with Xpath

I'm trying to retrieve data from a script tag that I'll attach below. From that script tag I would need the following data: digitalData.product.pvi_type_name , digitalData.product.pvi_subtype_name , digitalData.product.model_name, digitalData.product.displayName. I have written my own program in Python for retrieving but it doesn't work for now...

Script Tag Structure:

<script>
var COUNTRY_SHOP_STATUS = "buy";
var COUNTRY_SHOP_URL = "./buy";
var COUNTRY_WHERE_URL = "";
try {digitalData.page.pathIndicator.depth_2 = "mobile";} catch(e) {}
try {digitalData.page.pathIndicator.depth_3 = "mobile";} catch(e) {}
try {digitalData.page.pathIndicator.depth_4 = "smartphones";} catch(e) {}
try {digitalData.page.pathIndicator.depth_5 = "galaxy-note9";} catch(e) {}
try {digitalData.product.pvi_type_name      = "Mobile";} catch(e) {}
try {digitalData.product.pvi_subtype_name   = "Smartphone";} catch(e) {}
try {digitalData.product.model_name         = "SM-N960";} catch(e) {}
try {digitalData.product.displayName        = "galaxy note9";} catch(e) {}
try {digitalData.product.category           = digitalData.page.pathIndicator.depth_3;} catch(e) {}
</script>

Python Script:

import scrapy
import csv
import re

class QuotesSpider(scrapy.Spider):
name = "quotes"

def start_requests(self):
    with open('input.csv','r') as csvf:
        urlreader = csv.reader(csvf, delimiter=',',quotechar='"')
        for url in urlreader:
            if url[0]=="y":
                yield scrapy.Request(url[1])

def parse(self, response):
    def get_values(parameter, script):
        return re.findall('%s = "(.*)"' % parameter, script)[0]

    source_arr = response.xpath("//script[contains(., 'COUNTRY_SHOP_STATUS')]/text()").extract()
    if source_arr:
          source = source_arr[0]
          with open('output.csv', 'a',newline='') as csvfile:
              fieldnames = ['Category', 'Type', 'Model', 'SK']
              writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
              writer.writerow({'Category': get_values("pvi_type_name", source), 'Type': get_values("pvi_subtype_name", source), 'Model': get_values("pathIndicator.depth_5", source), 'SK': get_values("model_name", source)})

Upvotes: 0

Views: 55

Answers (1)

Andersson
Andersson

Reputation: 52665

If you got script content, try below to get required values:

import re

result = re.findall('product.*"(.*)"', source_arr[0])
print(result)
# ['Mobile', 'Smartphone', 'SM-N960', 'galaxy note9']

Upvotes: 1

Related Questions