Scrapy response have backslashes into element attributes

Question

I run the following code in a Scrapy Shell, to scrape data using a POST request:

url = 'http://www.ldg.co.uk/wp-admin/admin-ajax.php'

data = {'action': 'wpp_property_overview_pagination',
        'wpp_ajax_query[show_children]': 'true',
        'wpp_ajax_query[disable_wrapper]': 'true',
        'wpp_ajax_query[pagination]': 'off',
        'wpp_ajax_query[per_page]': '10',
        'wpp_ajax_query[query][property_category]': 'residential',
        'wpp_ajax_query[query][listing_type]': 'rent',
        'wpp_ajax_query[query][sort_by]': 'price_rent',
        'wpp_ajax_query[query][sort_order]': 'ASC',
        'wpp_ajax_query[query][pagi]': '0--10',
        'wpp_ajax_query[sorter]': '',
        'wpp_ajax_query[sort_by]': 'price_rent',
        'wpp_ajax_query[sort_order]': 'ASC',
        'wpp_ajax_query[template]': 'ajax',
        'wpp_ajax_query[requested_page]': '2'}

request = FormRequest(url, formdata = data)
fetch(request)

I know that inside the response are elements with the class "property-thumb", I've checked it by using Chrome Dev Tools, reading the response content. So, I try to scrape data using the XPath //*[@class="property-thumb"], this XPath is right (I use a Chrome plugin to check it with the content loaded into the page), but it isn't right if I try to use it from the Scrapy Shell:

In [10]: response.xpath('//*[@class="property-thumb"]')
Out[10]: []

I have noticed that response.body comes with a lot of backslashes, so I've figured out that the correct XPath should be //*[@class=\'\"property-thumb\"\']:

In [11]: response.xpath('//*[@class=\'\"property-thumb\"\']')
Out[11]: 
[\n      '>,
 \n      '>,
 \n      '>]

I think there is a problem with the way Scrapy manages strings from responses. Also, I think that those backslashes can generate more problems when scraping. Why do this happen? How can I solve it to use normal XPaths?

Padraic Cunningham · Accepted Answer

There is a very simple solution, you get json back not html:

url = 'http://www.ldg.co.uk/wp-admin/admin-ajax.php'

data = {'action': 'wpp_property_overview_pagination',
        'wpp_ajax_query[show_children]': 'true',
        'wpp_ajax_query[disable_wrapper]': 'true',
        'wpp_ajax_query[pagination]': 'off',
        'wpp_ajax_query[per_page]': '10',
        'wpp_ajax_query[query][property_category]': 'residential',
        'wpp_ajax_query[query][listing_type]': 'rent',
        'wpp_ajax_query[query][sort_by]': 'price_rent',
        'wpp_ajax_query[query][sort_order]': 'ASC',
        'wpp_ajax_query[query][pagi]': '0--10',
        'wpp_ajax_query[sorter]': '',
        'wpp_ajax_query[sort_by]': 'price_rent',
        'wpp_ajax_query[sort_order]': 'ASC',
        'wpp_ajax_query[template]': 'ajax',
        'wpp_ajax_query[requested_page]': '2'}
import requests
print(requests.post(url, data).json())

Which would give you:

{u'display': u'        
            
                
                    

                                    
            


            
                                    
                        
                            Lisson Street, Marylebone, London
                        
                    
                
                
                

                    
                                                    
                                
\xa3420/pw
                                                                    + fees
                                                            
                        
                    
                    
                                                    
                                
                                    
                                    1                                    Reception                                
                            
                        
                                                    
                                
                                    
                                    1                                    Bedroom                                
                            
                        
                                                    
                                
                                    
                                    1                                    Bathroom                                
                            
                                            



                        
                                                              View Floor Plan
      
                                                                                                |
                                                                Download Brochure
                                                    


                    
                        
                            View Details
                        
                    
                

            

        
            
            
                
                    

                                    
            


            
                                    
                        
                            Riding House Street, Fitzrovia, London
                        
                    
                
                
                

                    
                                                    
                                
\xa3425/pw
                                                                    + fees
                                                            
                        
                    
                    
                                                    
                                
                                    
                                    1                                    Reception                                
                            
                        
                                                    
                                
                                    
                                    1                                    Bedroom                                
                            
                        
                                                    
                                
                                    
                                    1                                    Bathroom                                
                            
                                            



                        
                                                              View Floor Plan
      
                                                                                                |
                                                                Download Brochure
                                                    


                    
                        
                            View Details
                        
                    
                

            

        
            
            
                
                    

                                    
            


            
                                    
                        
                            Grays Inn Road, Bloomsbury, London
                        
                    
                
                
                

                    
                                                    
                                
\xa3430/pw
                                                                    + fees
                                                            
                        
                    
                    
                                                    
                                
                                    
                                    1                                    Reception                                
                            
                        
                                                    
                                
                                    
                                    1                                    Bedroom                                
                            
                        
                                                    
                                
                                    
                                    1                                    Bathroom                                
                            
                                            



                        
                                                        
                                                                                            Download Brochure
                                                    


                    
                        
                            View Details
                        
                    
                

            

        
    ', u'wpp_query': {u'starting_row': 10, u'pagination': u'off', u'show_layout_toggle': False, u'current_page': u'2', u'requested_page': u'2', u'show_children': u'true', u'sortable_attrs': {u'menu_order': u'Default'}, u'sort_by': u'price_rent', u'sort_order': u'ASC', u'ajax_call': True, u'template': u'ajax', u'per_page': u'10', u'query': {u'sort_by': u'price_rent', u'pagi': u'10--10', u'listing_type': u'rent', u'sort_order': u'ASC', u'property_category': u'residential'}, u'sorter': u'', u'disable_wrapper': u'true', u'properties': {u'total': 60, u'results': [u'793240', u'836654', u'793035', u'793044', u'793078', u'793307', u'792965', u'793054', u'792811', u'793344']}`}}

The extra backslashes are there to escape the quotes etc.. Once you json.loads() the content the extra slashes so in your case call loads on the body:

 import json

 request = FormRequest(url, formdata = data)
 js = json.loads(fetch(request).body)

And to just get the html you would use the key html = js["display"].

Scrapy response have backslashes into element attributes

Answers (1)

Related Questions