CODEWITHSUNDEEP
xmlpython-3.x
Emved
Emved

Reputation: 25

Removing duplicates from XML with python

I have an automatically updated XML file with format:

<entry>
  <title>Movie 1</title>
  <summary>pics/movies/3a6f22.jpg</summary>
  <link>IMDB link</link>
  <time>Fri Dec  3 03:02:05 2018</time>
</entry>
<entry>
  <title>Movie 2</title>
  <summary>pics/movies/ae4r12.jpg</summary>
  <link>IMDB link</link>
  <time>Fri Dec  3 05:34:06 2018</time>
</entry>
<entry>
  <title>Movie 1</title>
  <summary>pics/movies/3a6f22.jpg</summary>
  <link>IMDB link</link>
  <time>Sat Dec  4 12:04:06 2018</time>
</entry>
<entry>
  <title>Movie 3</title>
  <summary>pics/movies/3f44j2.jpg</summary>
  <link>IMDB link</link>
  <time>Sat Dec 4  14:04:07 2018</time>
</entry>

My desired output would be:

<entry>
  <title>Movie 1</title>
  <summary>pics/movies/3a6f22.jpg</summary>
  <link>IMDB link</link>
  <time>Fri Dec  3 03:02:05 2018</time>
</entry>
<entry>
  <title>Movie 2</title>
  <summary>pics/movies/ae4r12.jpg</summary>
  <link>IMDB link</link>
  <time>Fri Dec  3 05:34:06 2018</time>
</entry>
<entry>
  <title>Movie 3</title>
  <summary>pics/movies/3f44j2.jpg</summary>
  <link>IMDB link</link>
  <time>Sat Dec 4  14:04:07 2018</time>
</entry>

That is being read by javascript and php to make a list with CSS. I am trying to filter out any duplicates (e.g. the last entry titled Movie 1). I've searched and found some xsl / xslt solutions that I couldn't get to function properly. My problem is that I would like to remove any duplicate entries with the same title, but the summary, link, or time do not need to match.

I have tried:

from lxml import etree

data = open('xmlparse.xsl')
xslt_content = data.read()
xslt_root = etree.XML(xslt_content)
dom = etree.parse("movies.old.xml")
transform = etree.XSLT(xslt_root)
result = transform(dom)
f = open('movies.new.xml', 'w')
f.write(str(result))
f.close()

which pulls from this .xsl

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
 <xsl:output omit-xml-declaration="yes" indent="yes"/>
 <xsl:strip-space elements="*"/>

 <xsl:param name="pDoc1" select="document('movies.old.xml')"/>
 <xsl:param name="pDoc2" select="document('movies.current.xml')"/>

 <xsl:template match="/">
  <xsl:apply-templates select="$pDoc1/*"/>
 </xsl:template>

 <xsl:template match="/*">
  <archive>
   <xsl:copy-of select="node()"/>
   <xsl:copy-of select="$pDoc2/*/entry[not(name = current()/*/name)]"/>
  </archive>
 </xsl:template>
</xsl:stylesheet>

no effect though, my new output file stays empty.

I have also tried using unique_everseen but that deletes data like <entry> and </entry>, rearranges time attributes to the end of the file, etc.. without mercy)

Upvotes: 2

Views: 1529

Answers (1)

Parfait
Parfait

Reputation: 107687

Consider using XSLT 1.0's grouping method of the Muenchian Method. Below script and demo assumes your root node is named root:

XSLT

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">

  <xsl:output method="xml" indent="yes"/>

  <xsl:key name="title_key" match="entry" use="title"/>

  <xsl:template match="/root">
      <xsl:copy>
        <xsl:apply-templates select="entry[generate-id() = 
                                           generate-id(key('title_key', title)[1])]"/>
      </xsl:copy>
  </xsl:template>

    <xsl:template match="entry">
        <xsl:copy>
            <xsl:copy-of select="*"/>
        </xsl:copy>
  </xsl:template>

</xsl:stylesheet>

XSLT Demo

Python

from lxml import etree

# LOAD XML AND XSLT
dom = etree.parse('movies_old.xml')
xsl = etree.parse('xslt_script.xsl')

# TRANSFORM XML
transform = etree.XSLT(xsl)
result = transform(dom)

# SAVE OUTPUT TO FILE
with open('movies_new.xml', 'wb') as f:
   f.write(result)

Upvotes: 1

Related Questions