CODEWITHSUNDEEP
pythonpandasdataframe
user1471980
user1471980

Reputation: 10626

how do you skip failed items from the api call using Python

I read from a csv file to populate a data frame. From this data frame, I go through each host:

 1. build an api
 2. request the api
 3. if api request is successful, I extract the data from the response json and build a data frame. 
  4. I keep going though each host and add the json ouput to my finalDF.
  5. if api request to a host unsuccessful, I need to skip that host.

my sample csv file is like this:

machine         OS          start_date  end_date
dbserver01      unix    1/23/2022   1/25/2022
dbserver101     unix    1/23/2022   1/25/2022
webserver101    unix    1/23/2022   1/25/2022
apache101       unix    1/23/2022   1/25/2022
appserver300    unix    1/23/2022   1/25/2022

I tried this:

import requests, json
import pandas as pd
import io
import warnings
import datetime
from datetime import datetime
import csv
import os.path


df = pd.read_csv(excel_path)

hosts=df['server'].dropna().tolist()
dataList = []
#declare finalDF to be appended
finalDF=pd.DataFrame(columns=['Metric', 'ProcId', 'TimeStamp', 'Value', 'Server', 'HostId', 'Proc_Name'])

try:
    for host in hosts:
        for id in env:
            hostIdUrl="example.com/+env+"/"+host"  
            resp = requests.get(hostIdUrl, verify=False,  headers={'Authorization': "Api-Token " + dnt_token[id]}).json()
            #print(resp)
            #print(hostIdUrl)
            if resp['totalCount'] > 0:
                dataList.append([host, id, dnt_env[id], dnt_token[id],resp['entities'][0]['entityId']])
            finalDF = finalDF.append(dataList)
    
    #print(finalDF)
    finalDF['TimeStamp']=pd.to_datetime(finalDF['TimeStamp']*1000000, utc=True).dt.tz_convert('EST').dt.strftime("%b-%d-%Y %H:%M:%S")
    #print(finalDF)
    finalDF=finalDF.drop_duplicates()
    date = datetime.now().strftime("%Y_%m_%d-%I_%M_%S_%p")
    finalDF.to_csv(output_path+"_"+date+'.csv', encoding='utf-8')
except:
    print("unable to get the host data")

The problem with this is that once there is a server that unable to get api for, script exists and all data is lost from the succeful api calls.

From the csv data, there may be servers where we cannot get api data for, we need to skip that and move on to the next. Any successful data need to be added to FinalDF data frame. I have tried Try: and Except: but I lose the whole data when meet a bad server. Any ideas?

Upvotes: 1

Views: 404

Answers (1)

jaaq
jaaq

Reputation: 1256

You simply need to change the order of your loops and put the except where it matters: around your request. Not sure if I understood you correctly so there's code for two versions in the except block.

finalDF = pd.DataFrame(...)

for env_id in envs:
  host_data = []
  for host in hosts:
    try:
       host_data.append(get_host_data(host, env_id))
    except Exception as e:
       print("cannot get data from host",host,"because","e")
       # if you need to just skip that one host do this
       continue
       # if you need to skip that env_id for all hosts do this instead
       host_data = []
       break
  for data in host_data:
    findalDF.append(data)

just put what you had in your inner for loop into a function called get_host_data and give it the relevant parameters(you don't need to I just used that to abbreviate the answer).

Upvotes: 1

Related Questions