CODEWITHSUNDEEP
pythonfile-iointeger
genclik27
genclik27

Reputation: 323

Storing a list, then reading it as integer

I have a question. It may be an easy one, but anyway I could not find a good idea. The question is that I have 2 python programs. First of them is giving 2 outputs, one of output is a huge list (like having thousands of another lists) and the other one is a simple csv file for the Weka. I need to store this list (first output) somehow to be able to use it as input of the other program later. I can not just send it to second program because when the first of the program is done, Weka should also produce new output for the second program. Hence, second program has to wait the outputs of first program and Weka. flow graph

The problem is that output list consists of lost of lists having numerical values. Simple example could be:

list1 = [[1,5,7],[14,3,27], [19,12,0], [23,8,17], [12,7]]

If I write this on a txt file, then when I try to read it, it takes all the values as string. Is there any easy and fast way (since data is big) to manage somehow taking all the values as integer? Or maybe in the first case, writing it as integer?

Upvotes: 1

Views: 121

Answers (2)

roman
roman

Reputation: 117380

I think this is good case to use pickle module

To save data:

import pickle

lst = [[1,5,7],[14,3,27], [19,12,0], [23,8,17], [12,7]]
pickle.dump(lst, open('data.pkl', 'wb'))

To read data from saved file:

import pickle

lst = pickle.load(open('data.pkl', 'r')

From documentation:

The pickle module implements a fundamental, but powerful algorithm for serializing and de-serializing a Python object structure. “Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation, whereby a byte stream is converted back into an object hierarchy. Pickling (and unpickling) is alternatively known as “serialization”, “marshalling,” [1] or “flattening”, however, to avoid confusion, the terms used here are “pickling” and “unpickling”.

there's also faster cPickle module:

To save data:

from cPickle import Pickler

p = Pickler(open('data2.pkl', 'wb'))
p.dump(lst)

To read data from saved file:

from cPickle import Unpickler

up = Unpickler(open('data.pkl', 'r'))
lst = up.load()

Upvotes: 1

bdeniker
bdeniker

Reputation: 1035

How about pickling the list output rather than outputting it as a plaintext representation? Have a look at the documentation for your version: it's basically a way to write Python objects to file, which you can then read from Python at any point to get identical objects.

Once you have the file open that you want to output to, the outputting difference will be quite minor, e.g.

import pickle

my_list = [[1, 2], [134, 76], [798, 5, 2]]
with open('outputfile.pkl', 'wb') as output:
    pickle.dump(my_list, output, -1)

And then just use the following way to read it in from your second program:

import pickle
my_list = pickle.load(open('outputfile.pkl', 'rb'))

Upvotes: 1

Related Questions