CODEWITHSUNDEEP
pythonpython-2.7csvbigdata
Rickson
Rickson

Reputation: 1090

Fastest way to export data of huge Python lists to a text file

I am searching for the most performant way to export the elements of up to ten Python lists [x1, x2, x3, ... xn], [y1, y2, y3, ... yn], [z1, z2, z3, ... zn], ... to a text file with a structure as follows:

x1 y1 z1  .  .  . 
x2 y2 z2  .  .  .
x3 y3 z3  .  .  .
 .  .  .  .  .  .
 .  .  .  .  .  .
 .  .  .  .  .  .
xn yn zn  .  .  .

What makes it challenging is that each list may have up to 1 million elements (only float or int numbers)

Any suggestions are highly appreciated.

Upvotes: 1

Views: 1956

Answers (2)

Jean-François Fabre
Jean-François Fabre

Reputation: 140307

Use the csv module and the writerows function to write the list of lists in one line.

Small standalone test:

import random,time


lists = [[random.randint(1,500) for _ in range(100000)] for _ in range(100)]

import csv
start_time=time.time()

with open("out.csv","w",newline="") as f:
    cw = csv.writer(f,delimiter=" ")
    cw.writerows(lists)

print(time.time()-start_time)

writes 100 lines of 100000 elements in 2 seconds on my machine (generating the list was slower than writing them back)

So you're just limited by the memory of your input list.

EDIT: this code above does not "transpose" properly so it's cheating. Using zip (python 3) does the trick directly using writerows so the code doesn't change much:

import random,time

n=1000000
list1 = list(range(1,n))
list2 = list(range(n+1,n*2))
list3 = list(range(2*n+1,n*3))

import csv
start_time=time.time()

with open("out.csv","w",newline="") as f:
    cw = csv.writer(f,delimiter=" ")
    cw.writerows(zip(list1,list2,list3))

print(time.time()-start_time)

for python2, use itertools.izip because zip returns a list: not memory-efficient. Python 2 compliant code:

import itertools
with open("out.csv","wb") as f:
    cw = csv.writer(f,delimiter=" ")
    cw.writerows(itertools.izip(list1,list2,list3))

If you have a list of lists:

list_of_lists = [list1,list2,list3]

you can use * to expand the list into arguments for zip or izip:

cw.writerows(zip(*lists_of_lists))

cw.writerows(itertools.izip(*lists_of_lists))

Upvotes: 5

Mohammad Yusuf
Mohammad Yusuf

Reputation: 17074

You can do something like this:

from itertools import izip
import csv

with open('new_file', 'w') as f:
    writer = csv.writer(f, delimiter=' ')
    for a in izip(l1, l2, ....., l10):
        writer.writerow(a)

Upvotes: 3

Related Questions