CODEWITHSUNDEEP
pythonxmlparsingbashopml
Sergei R.
Sergei R.

Reputation: 163

How do you extract feed urls from an OPML file exported from Google Reader?

I have a piece of software called Rss-Aware that I'm trying to use. It basically desktop feed-checker that checks if RSS feeds are updated and gives a notification through Ubuntu's Notify-OSD system.

However, to know what feeds to check, you have to list out the feed urls in a text file in ~/.rss-aware/rssfeeds.txt one after the other in a list with linebreak between each feed url. Something like:

http://example.com/feed.xml
http://othersite.org/feed.xml
http://othergreatsite.net/rss.xml

...Seems pretty simple right? Well, the list of feeds I'd like to use are exported from Google Reader as an OPML file (it's a type of XML) and I have no clue how to parse it to just output the the feed urls. It seems like it should be pretty straight forward yet I'm stumped.

I'd love if anyone could give an implementation in Python or Ruby or something I could do quickly from a prompt. A bash script would be awesome.

Thanks you so much for the help, I'm a really weak programmer and would love to learn how to do this basic parsing.

EDIT: Also, here is the OPML file I'm trying to extract the feed urls from.

Upvotes: 8

Views: 6807

Answers (5)

michael
michael

Reputation: 9799

There are a number of python packages that could help, this is one that is really old (as is this question itself), and likely no longer maintained (I can't even find the source code), but is quite simple to use. As a python one-liner (putting all python code onto the command-line):

$ pip install opml
$ python3 -c 'import opml; o=opml.parse("stitcher.opml"); print(*[x.xmlUrl for x in o], sep="\n")'

This prints out one URL per line from the OPML file. Alternatively just change the print statement as desired. Since the python package is not particularly useful beyond this, I'd uninstall it after you're done: pip uninstall opml (See: https://pypi.org/project/opml/ )

Upvotes: 0

Ash
Ash

Reputation: 3550

XML parsing was so easy to implement and worked great for me.

from xml.etree import ElementTree
def extract_rss_urls_from_opml(filename):
    urls = []
    with open(filename, 'rt') as f:
        tree = ElementTree.parse(f)
    for node in tree.findall('.//outline'):
        url = node.attrib.get('xmlUrl')
        if url:
            urls.append(url)
    return urls
urls = extract_rss_urls_from_opml('your_file')

Upvotes: 4

Daniel Cotter
Daniel Cotter

Reputation: 1464

You could also use a regex. I used the following search-and-replace regex to convert my Google Reader OPML export to a Firefox HTML live-bookmark import:

^\s+<outline.*?title="(.*?)".*?xmlUrl="(.*?)".*?htmlUrl="(.*?)".*?/>
<DT><A FEEDURL="$2" HREF="$3">$1</A>

Upvotes: 0

Kurt McKee
Kurt McKee

Reputation: 1410

I wrote a subscription list parser for this very purpose. It's called listparser, and it's written in Python. I just tested your OPML file, and it appears to parse the file perfectly. It will also make your feeds' labels available.

If you've ever used feedparser, the interface should be familiar:

>>> import listparser as lp
>>> d = lp.parse('https://dl.dropbox.com/u/670189/google-reader-subscriptions.xml')
>>> len(d.feeds)
112
>>> d.feeds[100].url
u'http://longreads.com/rss'
>>> d.feeds[100].tags
[u'reading']

It's possible to create the file with feed URLs using a script similar to:

import listparser as lp
d = lp.parse('https://dl.dropbox.com/u/670189/google-reader-subscriptions.xml')
f = open('/home/USERNAME/.rss-aware/rssfeeds.txt', 'w')
for i in d.feeds:
    f.write(i.url + '\n')
f.close()

Just replace USERNAME with your actual username. Done!

Upvotes: 5

Josh Rosen
Josh Rosen

Reputation: 13841

Since it's an XML file, you can use an XPath query to extract the urls. In the XML file, it looks like the rss feed urls are stored in xmlUrl attributes. The XPath expression //@xmlUrl will select all values of that attribute.

If you want to test this out in your web-browser, you can use an online XPath tester. If you want to perform this XPath query in Python, this question explains how to use XPath in Python. Additionally, the lxml docs have a page on using XPath in lxml that might be helpful.

Upvotes: 3

Related Questions