CODEWITHSUNDEEP
pythonseleniumxpathweb-scrapinggoogle-chrome-devtools
sam9750
sam9750

Reputation: 93

Using Xpath to scrape links and descriptions from this Etsy product listings page

I am trying to scrape all the links on https://www.etsy.com/market/happiness_bracelet and then the product descriptions from within each link extracted.

I am using a chrome extension called Scraper to input the xpath, obtained from right-clicking the element in Chrome Dev Tools. But not getting the desired result.

Problem: Can't find proper xpath for the links. What would be the proper setup to get the xpath of the links on that webpage and extract the product descriptions from within them?

Is there's a way to do it just using Chrome Dev Tools and the proper Xpath, or would I need Python/bs4/selenium for this task?

Thanks for your help.

Upvotes: 0

Views: 571

Answers (1)

Frank
Frank

Reputation: 871

With following XPath expression in Chrome Devtools you get all the p tag elements with the product descriptions: 

//div[contains(@class,'v2-listing-card__info')]/div/p[contains(@class,'body')]

See the image with the results:

enter image description here

Her you find the C# code to get the first line of text ('Handmade item') from the Overview paragraph for the first bracelet.

        // Go to the webpage
        Driver.Url = "https://www.etsy.com/market/happiness_bracelet";
        // Get all the links 
        IList<IWebElement> searchResults = Driver.FindElements(By.XPath("//a[contains(@class,'listing-link')]"));
        var i = 0;
        foreach (var result in searchResults)
        {
            i++;
            if (i > 5)
            // For all links except the first 5 hidden links
            {
                // Click the link to go to the page with information of the bracelet
                result.Click();
                // Write the first line of text of the Overview paragraph to the consol
                Console.WriteLine(Driver.FindElement(By.Id("item-overview")).FindElement(By.XPath(".//li")).Text);
                // More code needed here to pick the other information you needed 
            }             
        }

Upvotes: 1

Related Questions