CODEWITHSUNDEEP
pythonpython-2.7csv
bobsr
bobsr

Reputation: 3955

Python CSV file read and write with incremental number

There is Python CSV read file for input with 2 values. A incremental number and a name.

The records are in millions, most of the time stops in begining.

How to keep a log of incremental number on the write one, so it picks it up where it left off.

import csv

data = {}
with open("readfile.csv", "r") as f:
     for line in f.readlines():
        num,name = line.strip().split(',')
        data[num] = name


with open("output.csv", "wb") as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(["Number", "Name"])

Upvotes: 2

Views: 3466

Answers (2)

C Mars
C Mars

Reputation: 2979

One solution would be to use the Python file seek() and tell() functionality.

You can read the current position offset using tell and store this in another file.

You can then open the file and move to that position using seek.

example functionality would be:

import os

if not os.path.exists('readfile.csv'):
    with open("readfile.csv", "wb") as read_f:
        with open("position.dat", "wb") as pos_fi:



                read_f.write('aaa,111\n')
                read_f.write('bbb,222\n')
                read_f.write('ccc,333\n')
                read_f.write('ddd,444\n')

                pos_fi.write('0')


data = {}

with open('position.dat', 'rb') as pos_f:
    ps = pos_f.read()
    print 'Position is : ', ps
    p = int(ps)

# open your data file and the position file
with open('readfile.csv', 'rb') as read_f:

        # read the offset position and seek to that location
        read_f.seek(p)
        for line in iter(read_f.readline, ''):
                num,name = line.strip().split(',')
                print num, name
                data[num] = name

        position = str(read_f.tell())

# store your new offset position
with open("position.dat", "wb") as pos:
        pos.write(position)

EDIT:

This example now works.

  1. If you run the code once it will create the files.

  2. If you then edit the 'readfile.csv' and append more lines, and run the code again it again. It will pick up where it left off, and print out the new lines.

Please note to use seek(), you cannot then use readlines() directly on the file object. The trick is to wrap it in an iterator as above.

You would have to adapt your code around this, as I am not sure exactly what you are trying to read and write.

Yes, you could just open in append mode, to write to the end of the file.

Upvotes: 1

Joël
Joël

Reputation: 2822

The csvfile is a file object, so you can use seek and tell functions to move the cursor on the file. So, when you're done writing, you can get current position with pos = csvfile.tell(); then, at next opening, just do csvfile.seek(pos).

Note that you have to open your file in the same mode every time for this pos to be usable (binary maybe a good idea).

Another idea is simply to open csvfile in append mode: with open('output.csv', 'ab'). This writes at the end of the file.

Upvotes: 1

Related Questions