CODEWITHSUNDEEP
pythonxmlrss
soner
soner

Reputation: 133

Download and include referenced URL in XML

I have an RSS feed to a news source. Amongst the news text and other metadata, the feed also contains an URL reference to the comments section, which can also be in RSS format. I want to download and include the contents of the comments section for each news article. My aim is to create an RSS feed with the articles and the comments for each article included in the RSS, then convert this new RSS in calibre to PDF.

Here is an example XML:

<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <entry>
        <author>
            <name>Some Author</name>
            <uri>http://thenews.com</uri>
        </author>
        <category term="sports" label="Sports" />
        <content type="html">This is the news text.</content>
        <id>123abc</id>
        <link href="http://thenews.com/article/123abc/comments" />
        <updated>2016-04-29T13:44:00+00:00</updated>
        <title>The Title</title>
    </entry>
    <entry>
        <author>
            <name>Some other Author</name>
            <uri>http://thenews.com</uri>
        </author>
        <category term="sports" label="Sports" />
        <content type="html">This is another news text.</content>
        <id>123abd</id>
        <link href="http://thenews.com/article/123abd/comments" />
        <updated>2016-04-29T14:46:00+00:00</updated>
        <title>The other Title</title>
    </entry>
</feed>

Now I want to replace the <link href="http://thenews.com/article/123abc/comments" /> with the content of the URL. The RSS feed can be fetched by adding a /rss at the end of the URL. So in the end, a single entry would look like this:

<entry>
  <author>
    <name>Some Author</name>
    <uri>http://thenews.com</uri>
  </author>
  <category term="sports" label="Sports" />
  <content type="html">This is the news text.</content>
  <id>123abc</id>
  <comments>
    <comment>    
      <author>A commenter</author>
      <timestamp>2016-04-29T16:00:00+00:00</timestamp>
      <text>Cool story, yo!</text>
    </comment>
    <comment>
      <author>Another commenter</author>
      <timestamp>2016-04-29T16:01:00+00:00</timestamp>
      <text>This is interesting news.</text>
    </comment>
  </comments>
  <updated>2016-04-29T13:44:00+00:00</updated>
  <title>The Title</title>
</entry>

I'm open to any programming language. I tried this with python and lxml but couldn't get far. I was able to extract the comments URL and download the comments feed but couldn't replace the actual <link>-tag. Without having to download the actual RSS, here's how far I've come:

import lxml.etree as et
import urllib2
import re

# These will be downloaded from the RSS feed source when the code works
xmltext = """[The above news feed, too long to paste]"""
commentsRSS = """[The above comments feed]"""

hdr = { 'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'}

article = et.fromstring(xmltext)

for elem in article.xpath('//feed/entry'):
    commentsURL = elem.xpath('link/@href')

    #request  = urllib2.Request(commentsURL[0] + '.rss', headers=hdr)
    #comments = urllib2.urlopen(request).read()
    comments = commentsRSS

    # Now the <link>-tag should be replaced by the comments feed without the <?xml ...> tag

Upvotes: 0

Views: 54

Answers (1)

har07
har07

Reputation: 89305

For each <link> element, download XML from the href attribute and then parse the XML into a new Element. Then replace <link> with the corresponding new Element, something like this :

....
article = et.fromstring(xmltext)
ns = {'d': 'http://www.w3.org/2005/Atom'}
for elem in article.xpath('//d:feed/d:entry/d:link', namespaces=ns):
    request  = urllib2.Request(elem.attrib['href'] + '.rss', headers=hdr)
    comments = urllib2.urlopen(request).read()
    newElem = et.fromstring(comments)
    elem.getparent().replace(elem, newElem)

# print the result
print et.tostring(article)

Upvotes: 1

Related Questions