CODEWITHSUNDEEP
pythonpandasfuture-warning
JayFou
JayFou

Reputation: 41

Pandas concat function giving "FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated"

I have a pandas dataframe that contains a list of stock symbols. I want to loop through this list and call a function that calls an API to get more information about the individual stock symbol. I can only use the API for one symbol at a time and I need to end up with a new pandas dataset that contains all of the stock information because the data needs to be cleaned and I need all of the information together to do that.

My problem is that I receive a future warning for sometimes when I try and concatenate dataframes:

"FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation. pd_profiles = pd.concat([pd_profiles, pd_temp], axis=0)"

I have tried to figure this out and I've been adding in code to try and solve this issue but with no luck.

If anyone knows why I'm getting the future warning or if they know a better way of doing this, then please let me know.

def update_exchage_data(self):
   key = api_key
   # Here I create my list of equities that I get from an API but in the example I'm 
   # hard coding them in.  There are also other columns than the symbol but I wanted to 
   # keep the code simple.
   equities = [["APPL", "MSFT", "NVDA", "GOOG"] 
   pd_equities = pd.DataFrame(equities, columns=["symbol"]
   # Here I start create a list of columns that make up the final equities profile 
   # dataframe and create a blank dataframe that I will concatenate each returned 
   # profile to. 
   profile_columns = ["symbol", "name", "exchange", "sector", "industry"]
   pd_profiles = pd.DataFrame(columns=profile_columns)

   # This is an indicator to see if pd_profiles is empty in order to avoid 
   # concatenating an empty dataframe
   first_record = 1
   for index, row in pd_equities.iterrows():
   pd_temp = self.sub_get_profile(key, row["symbol"], profile_columns)
      #this is to check and see if the returned dataframe is empty
      if not pd_temp.empty and pd_temp.notnull and len(pd_temp) >= 1:
         if first_record == 1:
         first_record = 0
         pd_profiles = pd_temp
      else:
         pd_profiles = pd.concat([pd_profiles, pd_temp], axis=0)
pd_profiles = pd_profiles.reset_index(drop=True)


self.sub_get_profile(key, symbol, df_cols)
   # I didn't add any of the API code as it works fine and would just be taking up 
   # extra room.  The point is that it returns the data in a json called 
   # json_api_response

   # Here I create an empty dataframe so that even if the API call doesn't return 
   # anything, my function still returns an empty dataframe. 
   pd_data = pd.DataFrame(columns=df_cols)
      if not pd.DataFrame.from_dict(json_api_response, orient="columns").empty:
         pd_data = pd.DataFrame.from_dict(json_api_response, orient="columns")
return pd_data

Thanks in advance.

Upvotes: 4

Views: 4971

Answers (1)

msamedozmen
msamedozmen

Reputation: 1129

try to use

if not pd_temp.empty and pd_temp.notnull().any().any() and len(pd_temp) >= 1:

instead of

if not pd_temp.empty and pd_temp.notnull and len(pd_temp) >= 1:

Upvotes: 1

Related Questions