CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sqldatabricks
BdEngineer
BdEngineer

Reputation: 3179

"expression is neither present in the group by, nor is it an aggregate function" what is wrong here?

I am trying to apply a pivot on my dataframe as below

val pivot_company_model_vals_df =  company_model_vals_df.groupBy("company_id","id","date")
                          .pivot("code")
                          .agg( when( col("data_item_value_numeric").isNotNull,  
      first("numeric")).otherwise(first("value")) )

Error 

         org.apache.spark.sql.AnalysisException: expression '`data_item_value_numeric`' is neither present in the group by, nor is it an aggregate function.

Could you please help me what am I doing wrong here? Thank you

Upvotes: 0

Views: 7544

Answers (1)

BdEngineer
BdEngineer

Reputation: 3179

Issue fixed moving the first like below .agg( first(when: 

val pivot_company_model_vals_df =  company_model_vals_df.groupBy("company_id","model_id","data_date")
                          .pivot("code")
                          .agg( first(when( col("data_item_value_numeric").isNotNull,  
      col("numeric")).otherwise(col("_string")) ) )

Upvotes: 3

Related Questions