ValueError: could not broadcast input array from shape (5) into shape (7)

Question

In this code:

    import pandas as pd
    myj='{"columns":["tablename","alias_tablename","real_tablename","dbname","finalcost","columns","pri_col"],"index":[0,1],"data":[["b","b","vip_banners","openx","",["id","name","adlink","wap_link","ipad_link","iphone_link","android_link","pictitle","target","starttime","endtime","weight_limit","weight","introduct","isbutton","sex","tag","gomethod","showtype","version","warehouse","areaid","textpic","smallpicture","group","service_provider","channels","chstarttime","chendtime","tzstarttime","tzendtime","status","editetime","shownum","wap_version","ipad_version","iphone_version","android_version","showtime","template_id","app_name","acid","ab_test","ratio","ab_tset_type","acid_type","key_name","phone_models","androidpad_version","is_delete","ugep_group","author","content","rule_id","application_id","is_default","district","racing_id","public_field","editor","usp_expression","usp_group","usp_php_expression","is_pic_category","is_custom_finance","midwhitelist","is_freeshipping","resource_id","usp_property","always_display","pushtime","is_pmc","version_type","is_plan","loop_pic_frame_id","plan_personal_id","personal_id","is_img_auto","banner_type","ext_content"],"id"],["a","a","vip_adzoneassoc","openx","",["id","zone_id","ad_id","weight"],"id"]]}'
    df=pd.read_json(myj, orient='split')
    bl=['is_delete,status,author', 'endtime', 'banner_type', 'id', 'starttime', 'status,endtime','weight']
    al= ['zone_id,ad_id', 'zone_id,ad_id,id', 'ad_id', 'id', 'zone_id']
    # 
    #bl=['add_time', 'allot_time', 'create_time', 'end_pay_time', 'start_pay_time', 'order_status,update_time', 'order_type,order_status,add_time', 'order_type,order_status,end_pay_time', 'wms_flag,order_status,is_first,order_date', 'last_update_time', 'order_code', 'order_date', 'order_sn', 'parent_sn', 'id', 'user_id', 'wms_flag,order_date']
    #al=['area_id', 'last_update_time', 'mobile', 'parent_sn', 'id', 'transport_number', 'parent_sn']

    def get_index(row):

        print(row)
        if row.tablename=='b':
            return bl
        else:
            return al
    #        return ['is_delete,status,author', 'endtime', 'banner_type', 'id', 'starttime', 'status,endtime', 'weight']
    df['index_cols']=df.apply(get_index,axis=1)

I run into error:

ValueError: could not broadcast input array from shape (5) into shape (7)

Instead if I use the commented out bl and al evevything run fine. Also if I use

bl=['is_delete,status,author', 'endtime', 'banner_type', 'id', 'starttime', 'status,endtime']

it runs fine too, what 's the problem?

Answer 1

In pandas-0.22.0 a list coming from apply method could be used to construct a new dataframe, when its length equals the number of columns in the initial dataframe. For example:

>>> df = pd.DataFrame([range(100),range(100)])

    0   1   2   3   4   5   6   7   8   9   ...     90  91  92  93  94  95  96  97  98  99
0   0   1   2   3   4   5   6   7   8   9   ...     90  91  92  93  94  95  96  97  98  99
1   0   1   2   3   4   5   6   7   8   9   ...     90  91  92  93  94  95  96  97  98  99

You can return a list in apply and get a dataframe:

>>> df.apply(lambda x:(x+1).values.tolist(), axis=1)
    0   1   2   3   4   5   6   7   8   9   ...     90  91  92  93  94  95  96  97  98  99
0   1   2   3   4   5   6   7   8   9   10  ...     91  92  93  94  95  96  97  98  99  100
1   1   2   3   4   5   6   7   8   9   10  ...     91  92  93  94  95  96  97  98  99  100

but if length does not match dimension:

>>> df.apply(lambda x:(x+1).values.tolist()[:99], axis=1)

0    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14...
1    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14...

we get a Series.

And if you return lists of different dimension and the first one matches the dimension while the next one does not (like in your case) you get an error:

>>> df.apply(lambda x:[1] * 99 if x.name==0 else [0] * 100 , axis=1)
0    [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
1    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...

Works ok.

And this one

>>> df.apply(lambda x:[1] * 100 if x.name==0 else [0] * 99 , axis=1)

raises an error.

In pandas-0.23 you get a Series either way:

>>> df.apply(lambda x:(x+1).values.tolist(), axis=1)
0    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14...
1    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14...

>>> df.apply(lambda x:(x+1).values.tolist()[:9], axis=1)
0    [1, 2, 3, 4, 5, 6, 7, 8, 9]
1    [1, 2, 3, 4, 5, 6, 7, 8, 9]

This problem does not apply to tuples in pandas-0.22.0:

>>> df.apply(lambda x:(1,) * 9 if x.name==0 else (0,) * 10 , axis=1)
0       (1, 1, 1, 1, 1, 1, 1, 1, 1)
1    (0, 0, 0, 0, 0, 0, 0, 0, 0, 0)

You can use this fact in your case:

bl = ('is_delete,status,author', 'endtime', 'banner_type',
      'id', 'starttime', 'status,endtime', 'weight')
al = ('zone_id,ad_id', 'zone_id,ad_id,id', 'ad_id', 'id', 'zone_id')

>>> df.apply(get_index, axis=1)

0    (is_delete,status,author, endtime, banner_type...
1    (zone_id,ad_id, zone_id,ad_id,id, ad_id, id, z...
dtype: object