CODEWITHSUNDEEP
python-2.7pandas
TPPZ
TPPZ

Reputation: 4891

pandas shape issues when applying function returning multiple new columns

I need to return multiple calculated columns for each row of a pandas dataframe.

This error: ValueError: Shape of passed values is (4, 2), indices imply (4, 3) is raised when the apply function is executed in the following code snippet:

import pandas as pd

my_df = pd.DataFrame({
  'datetime_stuff': ['2012-01-20', '2012-02-16', '2012-06-19', '2012-12-15'],
  'url': ['http://www.something', 'http://www.somethingelse', 'http://www.foo', 'http://www.bar' ],
  'categories': [['foo', 'bar'], ['x', 'y', 'z'], ['xxx'], ['a123', 'a456']],   
})

my_df['datetime_stuff'] = pd.to_datetime(my_df['datetime_stuff'])
my_df.sort_values(['datetime_stuff'], inplace=True)

print(my_df.head())

def calculate_stuff(row):
  if row['url'].startswith('http'):
    categories = row['categories'] if type(row['categories']) == list else []
    calculated_column_x = row['url'] + '_other_stuff_'
  else:
    calculated_column_x = None
  another_column = 'deduction_from_fields'
  return calculated_column_x, another_column

print(my_df.shape)

my_df['calculated_column_x'], my_df['another_column'] = zip(*my_df.apply(calculate_stuff, axis=1))

Each row of the dataframe I am working on is more complicated than the example above, and the function calculate_stuff I am applying is using many different columns for each row, then returning multiple new columns.

However, the previous example still raises this ValueError related to the shape of the dataframe that I am not able to understand how to fix.

How to create multiple new columns (for each row) that can be calculated starting from the existing columns?

Upvotes: 2

Views: 1318

Answers (1)

piRSquared
piRSquared

Reputation: 294258

When you return a list or tuple from a function that is being applied, pandas attempts to shoehorn it back into the dataframe you ran apply over. Instead, return a series.

Reconfigured Code 

my_df = pd.DataFrame({
  'datetime_stuff': ['2012-01-20', '2012-02-16', '2012-06-19', '2012-12-15'],
  'url': ['http://www.something', 'http://www.somethingelse', 'http://www.foo', 'http://www.bar' ],
  'categories': [['foo', 'bar'], ['x', 'y', 'z'], ['xxx'], ['a123', 'a456']],   
})

my_df['datetime_stuff'] = pd.to_datetime(my_df['datetime_stuff'])
my_df.sort_values(['datetime_stuff'], inplace=True)

def calculate_stuff(row):
  if row['url'].startswith('http'):
    categories = row['categories'] if type(row['categories']) == list else []
    calculated_column_x = row['url'] + '_other_stuff_'
  else:
    calculated_column_x = None
  another_column = 'deduction_from_fields'

  # I changed this VVVV
  return pd.Series((calculated_column_x, another_column), ['calculated_column_x', 'another_column'])

my_df.join(my_df.apply(calculate_stuff, axis=1))

     categories datetime_stuff                       url                    calculated_column_x         another_column
0    [foo, bar]     2012-01-20      http://www.something      http://www.something_other_stuff_  deduction_from_fields
1     [x, y, z]     2012-02-16  http://www.somethingelse  http://www.somethingelse_other_stuff_  deduction_from_fields
2         [xxx]     2012-06-19            http://www.foo            http://www.foo_other_stuff_  deduction_from_fields
3  [a123, a456]     2012-12-15            http://www.bar            http://www.bar_other_stuff_  deduction_from_fields

Upvotes: 1

Related Questions