How to create a Pandas Pivot table?

Question

I have DataFrame of size (19999939, 5), now i am trying to create a pivot table for this using the following command:

movie_user_rating_pivot = user_rating.pivot(index = 'userId', columns = 'movieId', values = 'rating').fillna(0)

But this giving me Error message:

Can anyone help me?

Answer 1

I have found this issue: https://github.com/pandas-dev/pandas/issues/26314. The author says the issue doesn't occur for pandas 0.21. Theoretically you don't have that many cells (more than 2B), but maybe you got your dataframe size somehow wrong and it is actually bigger? I would recommend trying switching pandas version to 0.21 (if possible) and see what happens.

Answer 2

downgrade pandas to 21.1, it fixed my issue for now. Newer versions doesn't support large size