CODEWITHSUNDEEP
airflow
KristiLuna
KristiLuna

Reputation: 1903

using Airflow, from the same loop an API response with success continue to next task but a response w/ error has separate task that shows as failed

I'm wondering if there is a good way to go about doing this, I have the below loop that is making a list of all the Facebook pages that are throwing an error:

permission_error =[]
for idx, series in df.iterrows():
    id = series['id']
    access_token = series['access_token']
    response = requests.get(url=f'https://graph.facebook.com/{id}/feed?fields=insights.metric(post_impressions_unique,post_impressions)&since=2021-06-01&until=2021-06-20&access_token={access_token}')
    json_response = response.json()
    if 'error' in json_response:
        permission_error(series['name'])

what is the best way to break up the task so the pages that run successfully continue to the next task, but maybe something like a separate task that shows something other then 'success' with all the 'error' messages in the json_response. If so what's the best way to go about doing this?

Upvotes: 0

Views: 77

Answers (1)

Jarek Potiuk
Jarek Potiuk

Reputation: 20097

If the number of pages is small, you could return them as two separate dicts stored in XCom under different keys (via xcom.push() and pull different keys from xcom in two tasks that follow.

If the number of pages is in the order of 1000s, then the best will be to write those URLS out to an external storage (like GCS/S3) as two different objects and set names of those objects via those two XCom keys.

Upvotes: 2

Related Questions