CODEWITHSUNDEEP
javaxmlregexwebharvest
Ashwini
Ashwini

Reputation: 41

Using regex in web harvest xml

I'm using web harvest to scrap some e-commerce site.I'm iterating over the search page and getting each product details in output xml.But now I want to use regular expression in anchor(a) tag while scraping and get particular string.i.e.,

let $linktoprod :=data($item//a[@class="fk-anchor-link"]/@href)

The above line returns anchor tag href value of each product i.e., for first product the value returned is,

/casio-sheen-analog-watch-women/p/itmdaqmvzyy23hz5?pid=WATDAQMVVNQEM9CX&ref=6df83d8f-f61f-4648-b846-403938ae92fa

Now I want to using the regular expression like /([^/\?]+)\? and get the string between last / and ? i.e.,

itmdaqmvzyy23hz5

in the output xml. Please anyone who has any idea regarding this help me. Thank you.

Updated -

  <?xml version="1.0" encoding="UTF-8"?>

<config charset="ISO-8859-1">

<function name="download-multipage-list">
        <return>
            <while condition="${pageUrl.toString().length() != 0}" maxloops="${maxloops}" index="i">
                <empty>
                    <var-def name="content">
                        <html-to-xml>
                            <http url="${pageUrl}"/>
                        </html-to-xml>
                    </var-def>

                    <var-def name="nextLinkUrl">
                        <xpath expression="${nextXPath}">
                            <var name="content"/>
                        </xpath>
                    </var-def>

                    <var-def name="pageUrl">
                        <template>${sys.fullUrl(pageUrl.toString(), nextLinkUrl.toString())}</template>
                    </var-def>
                </empty>

                <xpath expression="${itemXPath}">
                 <var name="content"/>  
                </xpath>
            </while>
        </return>
    </function>

    <var-def name="products">          
        <call name="download-multipage-list">
                 <call-param name="pageUrl">http://www.flipkart.com/watches/pr?sid=reh%2Cr18</call-param>  
                 <call-param name="nextXPath">//a[starts-with(., 'Next')]/@href</call-param>
         <call-param name="itemXPath">//div[@class="product browse-product "]</call-param>
         <call-param name="pids"></call-param>
            <call-param name="maxloops">5</call-param>
         </call>

    </var-def>
   <var-def name="scrappedContent"> 
    <!-- iterates over all collected products and extract desired data -->

    <![CDATA[ <catalog> ]]> 

        <loop item="item" index="i">
            <list><var name="products"/></list>
            <body>
            <xquery>
                    <xq-param name="item" type="node()"><var name="item"/></xq-param>
                    <xq-expression><![CDATA[
                            declare variable $item as node() external;

                    let $linktoprod :=data($item//a[@class="fk-anchor-link"]/@href)
                  let $name := data($item//div[@class="title"])

                      return
                      <product>
                      <link>{$linktoprod}</link>
                    <title>{normalize-space($name)}</title>

                     </product>
                    ]]></xq-expression>
                </xquery>

            </body>
        </loop>
        <![CDATA[ </catalog> ]]>

 </var-def>  
</config>

My config xml is as show above.Where to use regexp code block in my xml? And I want the regexp to be applied to linktoprod and finally get the regexp output in link tag as output xml.Please anyone guide me. Thank you.

Upvotes: 0

Views: 1294

Answers (2)

Olaf Dietsche
Olaf Dietsche

Reputation: 74058

I don't know about web harvest, but if it supports a non greedy quantifier, you can use this pattern

/([^/]+?)\?

According to Web Harvest User manual - regexp you must insert something like this

<regexp>
    <regexp-pattern>/([^/]+?)\?</regexp-pattern>
    <regexp-source>
        /casio-sheen-analog-watch-women/p/itmdaqmvzyy23hz5?pid=WATDAQMVVNQEM9CX&amp;ref=6df83d8f-f61f-4648-b846-403938ae92fa
    </regexp-source>
    <regexp-result>
        <template>Last URL part is "${_1}"</template>
    </regexp-result>
</regexp>

In the <regexp-source> part you must insert your URL or variable to search for. Guessing from the manual and your config xml it might be something like

<regexp-source>
    <var>scrappedContent</var>
</regexp-source>

or

<regexp-source>
    ${linktoprod}
</regexp-source>

I think you must experiment a bit.

Upvotes: 1

Cerbrus
Cerbrus

Reputation: 72867

Try this regex:

/([^/]+)\?

You might need to strip the leading / and trailing ?.

To illustrate that the regex works, this is it's result in JavaScript:

var s = "/casio-sheen-analog-watch-women/p/itmdaqmvzyy23hz5?pid=WATDAQMVVNQEM9CX&amp;ref=6df83d8f-f61f-4648-b846-403938ae92fa"
console.log(s.match(/\/([^/]+)\?/g)); //  /itmdaqmvzyy23hz5?

Upvotes: 0

Related Questions