CODEWITHSUNDEEP
pythondata-processing
smatthewenglish
smatthewenglish

Reputation: 2889

Eliminate redundancies from a file using Python

How to condense, i.e. eliminate redundancies from, the following data: 

code: GB-ENG, jobs: 2673
code: GB-ENG, jobs: 23
code: GB-ENG, jobs: 459
code: GB-ENG, jobs: 346
code: RO-B, jobs: 9
code: DE-NW, jobs: 4
code: DE-BW, jobs: 3
code: DE-BY, jobs: 9
code: DE-HH, jobs: 34
code: DE-BY, jobs: 11
code: BE-BRU, jobs: 27
code: GB-ENG, jobs: 20

The output should be in this way:

GB-ENG, 3521
RO-B, 9
DE-NW, 4
DE-BW, 3
DE-HH, 34
DE-BY, 20
BE-BRU, 27

Described by 1 canonical representation of each code, i.e. DE-BY, that would represent the sum total aggregated over the numbers that are associated with each instance of that code, e.g.: 

code: DE-BY, jobs: 11
code: DE-BY, jobs: 9

becomes

DE-BY, 20

at the moment I'm creating that input with this Python script:

import json
import requests
from collections import defaultdict
from pprint import pprint

def hasNumbers(inputString):
    return any(char.isdigit() for char in inputString)

# open up the output of 'data-processing.py'
with open('job-numbers-by-location.txt') as data_file:

    # print the output to a file
    with open('phase_ii_output.txt', 'w') as output_file_:
        for line in data_file:
            identifier, name, coords, number_of_jobs = line.split("|")
            coords = coords[1:-1]
            lat, lng = coords.split(",")
            # print("lat: " + lat, "lng: " + lng)
            response = requests.get("http://api.geonames.org/countrySubdivisionJSON?lat="+lat+"&lng="+lng+"&username=s.matthew.english").json()


            codes = response.get('codes', [])
            for code in codes:
                if code.get('type') == 'ISO3166-2':
                    country_code = '{}-{}'.format(response.get('countryCode', 'UNKNOWN'), code.get('code', 'UNKNOWN'))
                    if not hasNumbers( country_code ):
                        # print("code: " + country_code + ", jobs: " + number_of_jobs)
                        output_file_.write("code: " + country_code + ", jobs: " + number_of_jobs)
    output_file_.close()

it would probably be most efficient to include this functionality as part of that script but I've not been able to yet figure out how.

Upvotes: 1

Views: 154

Answers (5)

Anas F
Anas F

Reputation: 76

assuming the text is stored in a text file, this would work 

infile = open('redundancy.txt','r')
a= infile.readlines()
print a
d={}
for item in a:
    c=item.strip('\n')    
    b=c.split()    
    if b[1] in d :
        d[b[1]]= int(d.get(b[1]))+eval((b[3]))
    else:
        d[b[1]]=b[3]
print d

it would give a result :

{'DE-BY,': 20, 'DE-HH,': '34', 'DE-BW,': '3', 'DE-NW,': '4', 'RO-B,': '9', 'GB-ENG,': 3521, 'BE-BRU,': '27'}

Upvotes: 1

Stefano
Stefano

Reputation: 453

This assume that you have a countries.txt formatted like 

code: GB-ENG jobs: 2673
code: GB-ENG jobs: 23
code: GB-ENG jobs: 459
code: GB-ENG jobs: 346
code: RO-B jobs: 9
code: DE-NW jobs: 4
code: DE-BW jobs: 3
code: DE-BY jobs: 9
code: DE-HH jobs: 34
code: DE-BY jobs: 11
code: BE-BRU jobs: 27
code: GB-ENG jobs: 20

Code Snippet

with open('countries.txt') as input_file, open('phase_ii_output.txt', 'w') as output_file:
            args = []
            dic = {}
            for line in input_file:
                args.append(line.split(" "))
            for n in args:
                key = n[1]
                num = int(n[3].rstrip())
                if key in dic:
                    dic[key] += num
                else:
                    dic[key] = num
            output_file.write(dic)

output

{'BE-BRU': 27, 'DE-BY': 20, 'DE-NW': 4, 'DE-BW': 3, 'RO-B': 9, 'GB-ENG': 3521, 'DE-HH': 34}

Upvotes: 1

Waxrat
Waxrat

Reputation: 2185

import sys, re
from collections import defaultdict
tally = defaultdict(int)
for line in sys.stdin:
    match = re.match(r'^code: (?P<code>\S+), jobs: (?P<jobs>\d+)', line).groupdict()
    tally[match["code"]] += int(match["jobs"])
for code, jobs in tally.iteritems():
    print "{}, {}".format(code, jobs)

Upvotes: 1

roganjosh
roganjosh

Reputation: 13185

The below code utilises the dict.get() method that you use throughout your current code to implement a counter. This is based on reading the values from your current .txt file but you could simply bypass the write to file and subsequent read using a similar method. 

tally = {}

with open('country_codes.txt', 'r') as infile, open('condensed.txt', 'w') as outfile:
    for line in infile:
        data = line.strip('\n')
        tag1, code, tag2, num = data.split()
        tally[code] = tally.get(code, 0) + int(num)
    for key, value in tally.items(): # Use .iteritems() for Python 2.x
        outfile.write(' '.join(map(str, [key, value, '\n'])))

This will take a file (country_codes.txt) with this structure:

code: GB-ENG, jobs: 2673
code: GB-ENG, jobs: 23
code: GB-ENG, jobs: 459
code: GB-ENG, jobs: 346
code: RO-B, jobs: 9
code: DE-NW, jobs: 4
code: DE-BW, jobs: 3
code: DE-BY, jobs: 9
code: DE-HH, jobs: 34
code: DE-BY, jobs: 11
code: BE-BRU, jobs: 27
code: GB-ENG, jobs: 20

And write this to condensed.txt as follows:

DE-BY, 20 
DE-HH, 34 
DE-BW, 3 
DE-NW, 4 
RO-B, 9 
GB-ENG, 3521 
BE-BRU, 27

Upvotes: 1

User2342342351
User2342342351

Reputation: 2174

You could do something like that:

data = """code: GB-ENG, jobs: 2673
code: GB-ENG, jobs: 23
code: GB-ENG, jobs: 459
code: GB-ENG, jobs: 346
code: RO-B, jobs: 9
code: DE-NW, jobs: 4
code: DE-BW, jobs: 3
code: DE-BY, jobs: 9
code: DE-HH, jobs: 34
code: DE-BY, jobs: 11
code: BE-BRU, jobs: 27
code: GB-ENG, jobs: 20"""


final_data = {}

for code, count in [_.strip('code: ').split(', jobs: ') for _ in data.split('\n')]:
    if code in final_data:
        final_data[code]['amount'] += int(count)

    else:
        final_data[code] = {'amount': int(count)}

for key, value in final_data.items():
    print('code: {}, jobs: {}'.format(key, value['amount']))

Upvotes: 1

Related Questions