CODEWITHSUNDEEP
pythonpython-3.x
mistermarko
mistermarko

Reputation: 305

How do I append a binary file?

This question relates to Project Euler, Problem 10: sum all the primes below two million. Before I used the right algorithm the program was very slow so I decided to write the primes it found to a data file in batches. Now I have the correct main algorithm but on principle I want to know how to append the data file.

#Project Euler 10a: list primes below 2,000,000.

import pickle

def numprime(x):
    for i in range(2, int(x**0.5)+1):
        if x % i == 0:
            return False
    else:
        return True

def growfile(primes, faccess):
    file = open('primelist2kk.dat', faccess)
    pickle.dump(primes, file)
    file.close()

def main(fchunk, faccess):
    #File input broken up for memory
    for i in range(10):
        plist = []
        for j in range(1, fchunk, 2):
            k = fchunk*i + j
            if numprime(k):
                plist.append(k)
        if i == 0:
            plist.pop(0)
            plist.insert(0,2)
        print()
        print(plist)
        growfile(plist, faccess)

def viewfile(faccess):
    g = open('primelist2kk.dat', faccess)  
    h = pickle.load(g)
    g.close()
    print(h)
    g.closed


#Supply 1/10 of the prime upper bound
#main(200, 'ab')
viewfile('rb')

I've tried every different sensible combination of file access codes I can think of but nothing works. Either subsequent data chunks delete the first one or they don't get saved at all. Any ideas?

NB Unhash main() to create the file and display the program output. Hash main() and unhash viewfile() to view the file.

Upvotes: 0

Views: 2669

Answers (2)

Blckknght
Blckknght

Reputation: 104722

You need to update your file using append mode (ab), but that's not enough. Each binary chunk of data that is written by pickle.dump will be independent. When you do a pickle.load on the file opened for reading, you'll only get the first one. You need to do a loop and keep loading data until there is no more left.

def viewfile():
    with open('primelist2kk.dat', 'rb') as g:
        try:
            while True: # run until there's an exception
                h = pickle.load(g)
                print(h)
        except EOFError: # this will be raised by load() when there's no more data
            pass

If you don't want each chuck to print separately, you could probably combine them in a single list.

Upvotes: 3

John La Rooy
John La Rooy

Reputation: 304245

You need append mode. ie.

faccess = 'ab'

Aside: Sieve_of_Eratosthenes is a faster way to generate those primes

example of the sieve

N = range(2000000)
P = []
for i in N:
    if i > 1:
        P.append(i)
        N[::i] = [0]*((len(N)+i-1)//i)

print P

Upvotes: 0

Related Questions