CODEWITHSUNDEEP
airflowdirected-acyclic-graphs
Adam
Adam

Reputation: 1754

Documentation for Airflow HTTP Operator/Sensor Extra Options?

I'm trying to read into the extra_options setting for Airflow to see what properties are possible to set (mainly interested in http timeout). I can't find any supporting documentation for this specific param anywhere: https://airflow.readthedocs.io/en/1.9.0/code.html?highlight=requests#airflow.operators.SimpleHttpOperator.

Has anyone worked with this before and is able to help?

Upvotes: 3

Views: 4707

Answers (2)

vurmux
vurmux

Reputation: 10030

According to source code (airflow.hooks.http_hook.HttpHook.run_and_check) extra_options uses these parameters:

        response = session.send(
            prepped_request,
            stream=extra_options.get("stream", False),
            verify=extra_options.get("verify", False),
            proxies=extra_options.get("proxies", {}),
            cert=extra_options.get("cert"),
            timeout=extra_options.get("timeout"),
            allow_redirects=extra_options.get("allow_redirects", True))

You can read more about them in requests library docs:

  • stream – (optional) whether to immediately download the response content. Defaults to False.
  • verify – (optional) Either a boolean, in which case it controls whether we verify the server’s TLS certificate, or a string, in which case it must be a path to a CA bundle to use. Defaults to True.
  • proxies – (optional) Dictionary mapping protocol or protocol and hostname to the URL of the proxy.
  • cert – (optional) if String, path to ssl client cert file (.pem). If Tuple, (‘cert’, ‘key’) pair.
  • timeout (float or tuple) – (optional) How long to wait for the server to send data before giving up, as a float, or a (connect timeout, read timeout) tuple.
  • allow_redirects (bool) – (optional) Set to True by default.

Upvotes: 5

y2k-shubham
y2k-shubham

Reputation: 11607

Following this trail of links in Airflow's source-code, you can easily determine what all things can be passed in SimpleHttpOperator, or more specifically, in extra field of Http Connection. I'm hereby adding trail of calls in Airflow's source that I used to trace the usage of extra_options

try:
    response = session.send(
        prepped_request,
        stream=extra_options.get("stream", False),
        verify=extra_options.get("verify", False),
        proxies=extra_options.get("proxies", {}),
        cert=extra_options.get("cert"),
        timeout=extra_options.get("timeout"),
        allow_redirects=extra_options.get("allow_redirects", True))

    if extra_options.get('check_response', True):
        self.check_response(response)
    return response

except requests.exceptions.ConnectionError as ex:
    self.log.warning(str(ex) + ' Tenacity will retry to execute the operation')
    raise ex

Upvotes: 2

Related Questions