python difflib comparing files

Question

I am trying to use difflib to produce diff for two text files containing tweets. Here is the code:

#!/usr/bin/env python

# difflib_test

import difflib

file1 = open('/home/saad/Code/test/new_tweets', 'r')
file2 = open('/home/saad/PTITVProgs', 'r')

diff = difflib.context_diff(file1.readlines(), file2.readlines())
delta = ''.join(diff)
print delta

Here is the PTITVProgs text file:

Watch PTI on April 6th (7) Dr Israr Shah at 10PM on Business Plus in "Talking Policy". Rgds #PTI
CORRECTION!! Watch PTI on April 6th (5) @Asad_Umar  at 8PM on ARY News. Rgds #PTI
Watch PTI on April 6th (5) @Asad_Umar  at 8PM on AAJ News (6) PTI vs PMLN at 8PM on NewsOne. Rgds #PTI
Watch PTI on April 6th (5) Asad Umar at 8PM on AAJ News (6) PTI vs PMLN at 8PM on NewsOne. Rgds #PTI
Watch PTI on April 6th (5) Waleed Iqbal at 8PM on Channel 5. Rgds #PTI
Watch PTI on April 6th (3) Dr Israr Shah at 10PM on PTV News. Rgds #PTI
Watch PTI on April 6th (4) Javed hashmi at 1PM on PTV News. Rgds #PTI
Watch PTI on April 6th (3) Imran Alvi at 1PM on AAJ News. Rgds #PTI
Watch PTI on April 6th (1) Dr @ArifAlvi, Andleeb Abbas and Ehtisham Ameer at 11PM on ARY News (2) Hamid Khan at 10PM on ATV. Rgds #PTI
Watch PTI on April 5th (1) Farooq Amjad Meer at 10:45PM on Dunya News. Rgds #PTI
Watch PTI on April 4th (4) Faisal Khan at 8PM on PTV News. Rgds #PTI
@FaisalJavedKhan
Watch PTI on April 4th (3) Faisal Khan at 11PM on ATV. Rgds #PTI
@FaisalJavedKhan
Watch PTI on April 4th (1) Dr Israr Shah at 8PM on Waqt News (2) Dr Arif Alvi at 9PM on PTV World. Rgds #PTI
@ArifAlvi
Watch PTI on April 3rd (12) Abrar ul Haq on 10PM on Dawn News (13) Shabbir Sial at 10PM on Channel5. Rgds #PTI
Watch PTI on April 3rd (11) Sadaqat Abbasi on 8PM on RohiTV. Rgds #PTI
Watch PTI on April 3rd (10) Dr Zarqa and Andleeb Abbas on 8PM on Waqt News. Rgds #PTI
Watch PTI on April 3rd (9) Fauzia Kasuri at 8PM on Din News. Rgds #PTI
Watch PTI on April 3rd (8) Mehmood Rasheed at 8PM on ARY News. Rgds #PTI
Watch PTI on April 3rd (7) Israr Abbasi (Repeat on Arp 4th) at 1:20AM and 1PM on Vibe TV. Rgds #PTI
Watch PTI on April 3rd (5) Rao Fahad at 9PM on Express News (6) Dr Seems Zia at 10:30PM on Health TV. Rgds #PTI

Here is the new_tweets text file:

Watch PTI on April 7th (3) Malaika Reza at 8PM on AAJ News (4) Shah Mehmood Qureshi at 8PM on Geo News. Rgds #PTI
Watch PTI on April 7th (2) Chairman IMRAN KHAN at 10PM on PTV News in News Night with Sadia Afzal, Rpt: 2AM, 2PM. Rgds #PTI
@ImranKhanPTI
Watch PTI on April 7th (1) Dr Waseem Shahzad NOW at 6PM on PTV News. Rgds #PTI
Watch PTI on April 6th (7) Dr Israr Shah at 10PM on Business Plus in "Talking Policy". Rgds #PTI
CORRECTION!! Watch PTI on April 6th (5) @Asad_Umar  at 8PM on ARY News. Rgds #PTI
Watch PTI on April 6th (5) @Asad_Umar  at 8PM on AAJ News (6) PTI vs PMLN at 8PM on NewsOne. Rgds #PTI
Watch PTI on April 6th (5) Asad Umar at 8PM on AAJ News (6) PTI vs PMLN at 8PM on NewsOne. Rgds #PTI
Watch PTI on April 6th (5) Waleed Iqbal at 8PM on Channel 5. Rgds #PTI
Watch PTI on April 6th (3) Dr Israr Shah at 10PM on PTV News. Rgds #PTI
Watch PTI on April 6th (4) Javed hashmi at 1PM on PTV News. Rgds #PTI
Watch PTI on April 6th (3) Imran Alvi at 1PM on AAJ News. Rgds #PTI
Watch PTI on April 6th (1) Dr @ArifAlvi, Andleeb Abbas and Ehtisham Ameer at 11PM on ARY News (2) Hamid Khan at 10PM on ATV. Rgds #PTI
Watch PTI on April 5th (1) Farooq Amjad Meer at 10:45PM on Dunya News. Rgds #PTI
Watch PTI on April 4th (4) Faisal Khan at 8PM on PTV News. Rgds #PTI
@FaisalJavedKhan
Watch PTI on April 4th (3) Faisal Khan at 11PM on ATV. Rgds #PTI
@FaisalJavedKhan
Watch PTI on April 4th (1) Dr Israr Shah at 8PM on Waqt News (2) Dr Arif Alvi at 9PM on PTV World. Rgds #PTI
@ArifAlvi
Watch PTI on April 3rd (12) Abrar ul Haq on 10PM on Dawn News (13) Shabbir Sial at 10PM on Channel5. Rgds #PTI
Watch PTI on April 3rd (11) Sadaqat Abbasi on 8PM on RohiTV. Rgds #PTI
Watch PTI on April 3rd (10) Dr Zarqa and Andleeb Abbas on 8PM on Waqt News. Rgds #PTI
Watch PTI on April 3rd (9) Fauzia Kasuri at 8PM on Din News. Rgds #PTI

Here is the diff i get from the program:

*** 
--- 
***************
*** 1,7 ****
- Watch PTI on April 7th (3) Malaika Reza at 8PM on AAJ News (4) Shah Mehmood Qureshi at 8PM on Geo News. Rgds #PTI
- Watch PTI on April 7th (2) Chairman IMRAN KHAN at 10PM on PTV News in News Night with Sadia Afzal, Rpt: 2AM, 2PM. Rgds #PTI
- @ImranKhanPTI
- Watch PTI on April 7th (1) Dr Waseem Shahzad NOW at 6PM on PTV News. Rgds #PTI
  Watch PTI on April 6th (7) Dr Israr Shah at 10PM on Business Plus in "Talking Policy". Rgds #PTI
  CORRECTION!! Watch PTI on April 6th (5) @Asad_Umar  at 8PM on ARY News. Rgds #PTI
  Watch PTI on April 6th (5) @Asad_Umar  at 8PM on AAJ News (6) PTI vs PMLN at 8PM on NewsOne. Rgds #PTI
--- 1,3 ----
***************
*** 21,24 ****
  Watch PTI on April 3rd (12) Abrar ul Haq on 10PM on Dawn News (13) Shabbir Sial at 10PM on Channel5. Rgds #PTI
  Watch PTI on April 3rd (11) Sadaqat Abbasi on 8PM on RohiTV. Rgds #PTI
  Watch PTI on April 3rd (10) Dr Zarqa and Andleeb Abbas on 8PM on Waqt News. Rgds #PTI
! Watch PTI on April 3rd (9) Fauzia Kasuri at 8PM on Din News. Rgds #PTI--- 17,23 ----
  Watch PTI on April 3rd (12) Abrar ul Haq on 10PM on Dawn News (13) Shabbir Sial at 10PM on Channel5. Rgds #PTI
  Watch PTI on April 3rd (11) Sadaqat Abbasi on 8PM on RohiTV. Rgds #PTI
  Watch PTI on April 3rd (10) Dr Zarqa and Andleeb Abbas on 8PM on Waqt News. Rgds #PTI
! Watch PTI on April 3rd (9) Fauzia Kasuri at 8PM on Din News. Rgds #PTI
! Watch PTI on April 3rd (8) Mehmood Rasheed at 8PM on ARY News. Rgds #PTI
! Watch PTI on April 3rd (7) Israr Abbasi (Repeat on Arp 4th) at 1:20AM and 1PM on Vibe TV. Rgds #PTI
! Watch PTI on April 3rd (5) Rao Fahad at 9PM on Express News (6) Dr Seems Zia at 10:30PM on Health TV. Rgds #PTI

As you can see by quickly comparing the two source files (PTITVProgs and new_tweets) that the difference between them are the 3 tweets that were on April 7th and 3 tweets on April 3rd.

I just want the lines in new_tweets that are not in PTITVProgs to appear in the diff.

But it throws a bunch of text that I don't want to see. I don't know what *** 1,7*** and *** 1,3*** in the diff output stand for...? What is the proper way to get only the changed lines?

Answer 1

One of the best tools in Python to compare files and highlight differences is Google's diff-match-patch library.

Let's see it for your example.

PTITVProgs = """Watch PTI on April 6th (7) Dr Israr Shah at 10PM on Business Plus in "Talking Policy". Rgds #PTI
CORRECTION!! Watch PTI on April 6th (5) @Asad_Umar  at 8PM on ARY News. Rgds #PTI
Watch PTI on April 6th (5) @Asad_Umar  at 8PM on AAJ News (6) PTI vs PMLN at 8PM on NewsOne. Rgds #PTI
Watch PTI on April 6th (5) Asad Umar at 8PM on AAJ News (6) PTI vs PMLN at 8PM on NewsOne. Rgds #PTI
Watch PTI on April 6th (5) Waleed Iqbal at 8PM on Channel 5. Rgds #PTI
Watch PTI on April 6th (3) Dr Israr Shah at 10PM on PTV News. Rgds #PTI
Watch PTI on April 6th (4) Javed hashmi at 1PM on PTV News. Rgds #PTI
Watch PTI on April 6th (3) Imran Alvi at 1PM on AAJ News. Rgds #PTI
Watch PTI on April 6th (1) Dr @ArifAlvi, Andleeb Abbas and Ehtisham Ameer at 11PM on ARY News (2) Hamid Khan at 10PM on ATV. Rgds #PTI
Watch PTI on April 5th (1) Farooq Amjad Meer at 10:45PM on Dunya News. Rgds #PTI
Watch PTI on April 4th (4) Faisal Khan at 8PM on PTV News. Rgds #PTI
@FaisalJavedKhan
Watch PTI on April 4th (3) Faisal Khan at 11PM on ATV. Rgds #PTI
@FaisalJavedKhan
Watch PTI on April 4th (1) Dr Israr Shah at 8PM on Waqt News (2) Dr Arif Alvi at 9PM on PTV World. Rgds #PTI
@ArifAlvi
Watch PTI on April 3rd (12) Abrar ul Haq on 10PM on Dawn News (13) Shabbir Sial at 10PM on Channel5. Rgds #PTI
Watch PTI on April 3rd (11) Sadaqat Abbasi on 8PM on RohiTV. Rgds #PTI
Watch PTI on April 3rd (10) Dr Zarqa and Andleeb Abbas on 8PM on Waqt News. Rgds #PTI
Watch PTI on April 3rd (9) Fauzia Kasuri at 8PM on Din News. Rgds #PTI
Watch PTI on April 3rd (8) Mehmood Rasheed at 8PM on ARY News. Rgds #PTI
Watch PTI on April 3rd (7) Israr Abbasi (Repeat on Arp 4th) at 1:20AM and 1PM on Vibe TV. Rgds #PTI
Watch PTI on April 3rd (5) Rao Fahad at 9PM on Express News (6) Dr Seems Zia at 10:30PM on Health TV. Rgds #PTI"""

new_tweets = """Watch PTI on April 7th (3) Malaika Reza at 8PM on AAJ News (4) Shah Mehmood Qureshi at 8PM on Geo News. Rgds #PTI
Watch PTI on April 7th (2) Chairman IMRAN KHAN at 10PM on PTV News in News Night with Sadia Afzal, Rpt: 2AM, 2PM. Rgds #PTI
@ImranKhanPTI
Watch PTI on April 7th (1) Dr Waseem Shahzad NOW at 6PM on PTV News. Rgds #PTI
Watch PTI on April 6th (7) Dr Israr Shah at 10PM on Business Plus in "Talking Policy". Rgds #PTI
CORRECTION!! Watch PTI on April 6th (5) @Asad_Umar  at 8PM on ARY News. Rgds #PTI
Watch PTI on April 6th (5) @Asad_Umar  at 8PM on AAJ News (6) PTI vs PMLN at 8PM on NewsOne. Rgds #PTI
Watch PTI on April 6th (5) Asad Umar at 8PM on AAJ News (6) PTI vs PMLN at 8PM on NewsOne. Rgds #PTI
Watch PTI on April 6th (5) Waleed Iqbal at 8PM on Channel 5. Rgds #PTI
Watch PTI on April 6th (3) Dr Israr Shah at 10PM on PTV News. Rgds #PTI
Watch PTI on April 6th (4) Javed hashmi at 1PM on PTV News. Rgds #PTI
Watch PTI on April 6th (3) Imran Alvi at 1PM on AAJ News. Rgds #PTI
Watch PTI on April 6th (1) Dr @ArifAlvi, Andleeb Abbas and Ehtisham Ameer at 11PM on ARY News (2) Hamid Khan at 10PM on ATV. Rgds #PTI
Watch PTI on April 5th (1) Farooq Amjad Meer at 10:45PM on Dunya News. Rgds #PTI
Watch PTI on April 4th (4) Faisal Khan at 8PM on PTV News. Rgds #PTI
@FaisalJavedKhan
Watch PTI on April 4th (3) Faisal Khan at 11PM on ATV. Rgds #PTI
@FaisalJavedKhan
Watch PTI on April 4th (1) Dr Israr Shah at 8PM on Waqt News (2) Dr Arif Alvi at 9PM on PTV World. Rgds #PTI
@ArifAlvi
Watch PTI on April 3rd (12) Abrar ul Haq on 10PM on Dawn News (13) Shabbir Sial at 10PM on Channel5. Rgds #PTI
Watch PTI on April 3rd (11) Sadaqat Abbasi on 8PM on RohiTV. Rgds #PTI
Watch PTI on April 3rd (10) Dr Zarqa and Andleeb Abbas on 8PM on Waqt News. Rgds #PTI
Watch PTI on April 3rd (9) Fauzia Kasuri at 8PM on Din News. Rgds #PTI"""

To find the differences

import diff_match_patch as dmp_module    # !pip install diff-match-patch
dmp = dmp_module.diff_match_patch()
diff = dmp.diff_main(PTITVProgs, new_tweets)
dmp.diff_cleanupSemantic(diff)
# diff
# [(1,
#   'Watch PTI on April 7th (3) Malaika Reza at 8PM on AAJ News (4) Shah Mehmood Qureshi at 8PM on Geo News. Rgds #PTI\nWatch PTI on April 7th (2) Chairman IMRAN KHAN at 10PM on PTV News in News Night with Sadia Afzal, Rpt: 2AM, 2PM. Rgds #PTI\n@ImranKhanPTI\nWatch PTI on April 7th (1) Dr Waseem Shahzad NOW at 6PM on PTV News. Rgds #PTI\n'),
#  (0,
#   'Watch PTI on April 6th (7) Dr Israr Shah at 10PM on Business Plus in "Talking Policy". Rgds #PTI\nCORRECTION!! Watch PTI on April 6th (5) @Asad_Umar  at 8PM on ARY News. Rgds #PTI\nWatch PTI on April 6th (5) @Asad_Umar  at 8PM on AAJ News (6) PTI vs PMLN at 8PM on NewsOne. Rgds #PTI\nWatch PTI on April 6th (5) Asad Umar at 8PM on AAJ News (6) PTI vs PMLN at 8PM on NewsOne. Rgds #PTI\nWatch PTI on April 6th (5) Waleed Iqbal at 8PM on Channel 5. Rgds #PTI\nWatch PTI on April 6th (3) Dr Israr Shah at 10PM on PTV News. Rgds #PTI\nWatch PTI on April 6th (4) Javed hashmi at 1PM on PTV News. Rgds #PTI\nWatch PTI on April 6th (3) Imran Alvi at 1PM on AAJ News. Rgds #PTI\nWatch PTI on April 6th (1) Dr @ArifAlvi, Andleeb Abbas and Ehtisham Ameer at 11PM on ARY News (2) Hamid Khan at 10PM on ATV. Rgds #PTI\nWatch PTI on April 5th (1) Farooq Amjad Meer at 10:45PM on Dunya News. Rgds #PTI\nWatch PTI on April 4th (4) Faisal Khan at 8PM on PTV News. Rgds #PTI\n@FaisalJavedKhan\nWatch PTI on April 4th (3) Faisal Khan at 11PM on ATV. Rgds #PTI\n@FaisalJavedKhan\nWatch PTI on April 4th (1) Dr Israr Shah at 8PM on Waqt News (2) Dr Arif Alvi at 9PM on PTV World. Rgds #PTI\n@ArifAlvi\nWatch PTI on April 3rd (12) Abrar ul Haq on 10PM on Dawn News (13) Shabbir Sial at 10PM on Channel5. Rgds #PTI\nWatch PTI on April 3rd (11) Sadaqat Abbasi on 8PM on RohiTV. Rgds #PTI\nWatch PTI on April 3rd (10) Dr Zarqa and Andleeb Abbas on 8PM on Waqt News. Rgds #PTI\nWatch PTI on April 3rd (9) Fauzia Kasuri at 8PM on Din News. Rgds #PTI'),
#  (-1,
#   '\nWatch PTI on April 3rd (8) Mehmood Rasheed at 8PM on ARY News. Rgds #PTI\nWatch PTI on April 3rd (7) Israr Abbasi (Repeat on Arp 4th) at 1:20AM and 1PM on Vibe TV. Rgds #PTI\nWatch PTI on April 3rd (5) Rao Fahad at 9PM on Express News (6) Dr Seems Zia at 10:30PM on Health TV. Rgds #PTI')]

In the output list diff, the first variable is 0/1/-1 and second variable is a text. 0 which means the text is common, 1 means text is added in the second file, -1 means text is removed from the first file. So, to see lines in new_tweets that are not in PTITVProgs:

for change,text in diff:
    if change==1:
        print(text)
# Watch PTI on April 7th (3) Malaika Reza at 8PM on AAJ News (4) Shah Mehmood Qureshi at 8PM on Geo News. Rgds #PTI
# Watch PTI on April 7th (2) Chairman IMRAN KHAN at 10PM on PTV News in News Night with Sadia Afzal, Rpt: 2AM, 2PM. Rgds #PTI
# @ImranKhanPTI
# Watch PTI on April 7th (1) Dr Waseem Shahzad NOW at 6PM on PTV News. Rgds #PTI

Please note it works well even if the lines are partially different.

But, I use it by visualizing an HTML of differences:

html = dmp.diff_prettyHtml(diff)
html = html.replace('¶', '')    # Not displaying newline characters. Newline differences won't appear.

# Display it in Jupyter
from IPython.core.display import display, HTML
display(HTML(html))

Red strikethroughs are removed from the first file, and green underlineds are added in the second file.

Answer 2

There are multiple diff styles and different functions exist for them in the difflib library. unified_diff, ndiff and context_diff.

If you don't want the line number summaries, ndiff function gives a Differ-style delta:

import difflib

f1 = '''1
2
3
4
5'''
f2 = '''1
3
4
5
6'''

diff = difflib.ndiff(f1,f2)

for l in diff:
    print(l)

Output:

  1
- 2          
  3          
  4          
  5   
+ 6

EDIT:

You could also parse the diff to extract only the changes if that's what you want:

>>>changes = [l for l in diff if l.startswith('+ ') or l.startswith('- ')]

>>>for c in changes:
       print(c)
>>>
- 2
+ 6

Answer 3

Just parse output of diff like this (change '- ' to '+ ' if needed):

#!/usr/bin/env python

# difflib_test

import difflib

file1 = open('/home/saad/Code/test/new_tweets', 'r')
file2 = open('/home/saad/PTITVProgs', 'r')

diff = difflib.ndiff(file1.readlines(), file2.readlines())
delta = ''.join(x[2:] for x in diff if x.startswith('- '))
print delta