CODEWITHSUNDEEP
pythonlistpandasdataframeseries
Brian
Brian

Reputation: 1645

How do I turn a dataframe into a series of lists?

I have had to do this several times and I'm always frustrated. I have a dataframe:

df = pd.DataFrame([[1, 2, 3, 4], [5, 6, 7, 8]], ['a', 'b'], ['A', 'B', 'C', 'D'])

print df

   A  B  C  D
a  1  2  3  4
b  5  6  7  8

I want to turn df into:

pd.Series([[1, 2, 3, 4], [5, 6, 7, 8]], ['a', 'b'])

a    [1, 2, 3, 4]
b    [5, 6, 7, 8]
dtype: object

I've tried

df.apply(list, axis=1)

Which just gets me back the same df

What is a convenient/effective way to do this?

Upvotes: 24

Views: 4413

Answers (3)

Sayali Sonawane
Sayali Sonawane

Reputation: 12609

Dataframe to list conversion

List_name =df_name.values.tolist()

Upvotes: 0

piRSquared
piRSquared

Reputation: 294556

pandas tries really hard to make making dataframes convenient. As such, it interprets lists and arrays as things you'd want to split into columns. I'm not going to complain, this is almost always helpful.

I've done this one of two ways.

Option 1:

# Only works with a non MultiIndex
# and its slow, so don't use it
df.T.apply(tuple).apply(list)

Option 2:

pd.Series(df.T.to_dict('list'))

Both give you:

a    [1, 2, 3, 4]
b    [5, 6, 7, 8]
dtype: object

However Option 2 scales better.

Timing

given df

enter image description here

much larger df

from string import ascii_letters
letters = list(ascii_letters)
df = pd.DataFrame(np.random.choice(range(10), (52 ** 2, 52)),
                  pd.MultiIndex.from_product([letters, letters]),
                  letters)

Results for df.T.apply(tuple).apply(list) are erroneous because that solution doesn't work over a MultiIndex.

enter image description here

Upvotes: 9

jezrael
jezrael

Reputation: 863751

You can first convert DataFrame to numpy array by values, then convert to list and last create new Series with index from df if need faster solution:

print (pd.Series(df.values.tolist(), index=df.index))
a    [1, 2, 3, 4]
b    [5, 6, 7, 8]
dtype: object

Timings with small DataFrame:

In [76]: %timeit (pd.Series(df.values.tolist(), index=df.index))
1000 loops, best of 3: 295 µs per loop

In [77]: %timeit pd.Series(df.T.to_dict('list'))
1000 loops, best of 3: 685 µs per loop

In [78]: %timeit df.T.apply(tuple).apply(list)
1000 loops, best of 3: 958 µs per loop

and with large:

from string import ascii_letters
letters = list(ascii_letters)
df = pd.DataFrame(np.random.choice(range(10), (52 ** 2, 52)),
                  pd.MultiIndex.from_product([letters, letters]),
                  letters)

In [71]: %timeit (pd.Series(df.values.tolist(), index=df.index))
100 loops, best of 3: 2.06 ms per loop

In [72]: %timeit pd.Series(df.T.to_dict('list'))
1 loop, best of 3: 203 ms per loop

In [73]: %timeit df.T.apply(tuple).apply(list)
1 loop, best of 3: 506 ms per loop

Upvotes: 22

Related Questions