CODEWITHSUNDEEP
pythonarrayslistnumpy
zell
zell

Reputation: 10204

Efficient data structure for storing N lists where N is very large

I will need to store N lists, where N is large (1 million). For example,

[2,3]
[4,5,6]
...
[4,5,6,7]

Each item is a list of about 0-10000 elements. I wanted to use a numpy array of lists, like

np.array([[2,3],[4,5,6])

Then I got efficiency issues when trying to append to the lists in the numpy array. Also I was told here: Efficiently append an element to each of the lists in a large numpy array, to not use numpy array of lists.

What would be a good data structure for storing such data, in terms of memory and time efficiency?

Upvotes: 0

Views: 851

Answers (4)

IoaTzimas
IoaTzimas

Reputation: 10624

Maybe use a dictionary:

d={}
for i in range(N):
  d[i]=your_nth_list

And you will simply append them by:

d[k].append(additional_items)

(It's efficient for 10.000.000 lists of 1000 items each)

Upvotes: 3

aarribas12
aarribas12

Reputation: 574

Unless the elements youre storing follow some pattern you must use nested list since there is no other way to get those elements out of the others.

In Python:

listOfLists = [[1,2,3],
               [4,5,6],
               [7,8,9]]

So whenever you want to operate with this list you can use numpy functions

>>> np.mean(listOfLists)
5.0
>>> np.max(listOfLists)
9

Upvotes: 1

NHL
NHL

Reputation: 287

You could use nested lists but they are not efficent in terms of complexity. In fact, it is linear, you could use dictionaries to get better results :

dict={}
for i in range(numer_of_lists) :
  dict[str(i)]=your_i-th_list

Then access the i-th element withdict[str(i)] Then, appening an element will be as easy

`

Upvotes: 1

Divyessh
Divyessh

Reputation: 2721

try nested list

nestedList = [[2,3],[4,5,6]]

Upvotes: 1

Related Questions