CODEWITHSUNDEEP
javaairflowopenapi
carlosU
carlosU

Reputation: 41

Apache Airflow 2.0.0 Trigger a DAG Run via the Stable REST API fails with FORBIDDEN

Apache Airflow version:

2.0.0

Environment:

What happened:

I am trying to trigger a DAG run via the new stable REST-API 1.0.0 by using a Java-based client. The Java client is generated from the Airflow OpenAPI specification with OpenAPI generator version 5.0.0 .

Custom configurations in my airflow.cfg:

web_server_host = 0.0.0.0
web_server_port = 8080
auth_backend = airflow.api.auth.backend.default

In addition I have created one admin user:

airflow users create \
    --username admin \
    --password secret \
    --firstname john \
    --lastname doe \
    --role Admin \
    --email [email protected]

In Airflow I have one active DAG demo

I trigger the DAG execution via the java client with the following code snippet:

ApiClient defaultClient = Configuration.getDefaultApiClient();
defaultClient.setBasePath("http://1.2.3.4:8080/api/v1);
DagRunApi dagRunApi = new DagRunApi(defaultClient);
DAGRun dagRun = new DAGRun();
try {
      DAGRun result = dagRunApi.postDagRun("demo", dagRun);

      LOGGER.debug(result);

    } catch (ApiException e) {
      e.printStackTrace();
    }

However, I receive the following exception:

org.openapitools.client.ApiException: FORBIDDEN
    at org.openapitools.client.ApiClient.handleResponse(ApiClient.java:1012)
    at org.openapitools.client.ApiClient.execute(ApiClient.java:925)
    at org.openapitools.client.api.DagRunApi.postDagRunWithHttpInfo(DagRunApi.java:740)
    at org.openapitools.client.api.DagRunApi.postDagRun(DagRunApi.java:715)
    at de.baas.evaluation.scheduler.airflow.ExternalTrigger.triggerDAGRun(ExternalTrigger.java:46)
    at de.baas.evaluation.scheduler.service.BaaSScheduler$1.message(BaaSScheduler.java:60)
    at de.baas.evaluation.scheduler.service.BaaSScheduler$1.message(BaaSScheduler.java:53)
    at io.lettuce.core.pubsub.PubSubEndpoint.notifyListeners(PubSubEndpoint.java:217)
    at io.lettuce.core.pubsub.PubSubEndpoint.notifyMessage(PubSubEndpoint.java:206)
    at io.lettuce.core.pubsub.PubSubCommandHandler.doNotifyMessage(PubSubCommandHandler.java:292)
    at io.lettuce.core.pubsub.PubSubCommandHandler.notifyPushListeners(PubSubCommandHandler.java:223)
    at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:606)
    at io.lettuce.core.pubsub.PubSubCommandHandler.decode(PubSubCommandHandler.java:112)
    at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:560)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

The Airflow logs do not show any error message.

What you expected to happen:

Previously, I also the outdated experimental REST-API to trigger tasks externally (without a client but using custom REST calls) and it worked without issues.

With the new stable API it seems that my client does not have sufficient permissions even if the authentication is deactivated via airflow.api.auth.backend.default

But also adding username and password to the client creation via

defaultClient.setUsername("admin");
defaultClient.setPassword("secret");

results in the same error: FORBIDDEN

Do I need to create a dedicated REST-API user and if yes is there any documentation on this topic?

How to reproduce it:

Create the Java client (probably any client will produce this error) via OpenAPI generator:

  1. java -jar codegen/openapi-generator-cli-5.0.0.jar generate -i openapi.yaml -g java -o .
  2. create a simple java class and apply the code snippets from above

Upvotes: 1

Views: 1725

Answers (1)

carlosU
carlosU

Reputation: 41

In case anyone else runs into the issue, I was able to resolve it as follows:

  1. change the auth backend to auth_backend = airflow.api.auth.backend.basic_auth because airflow.api.auth.backend.default does not work with the new REST-API

  2. apply the following change to the Airflow openapi.yaml which is required to generate the client code corretly (otherwise username/password) are not transmitted corretly, see also: https://github.com/apache/airflow/issues/13739

# The API will provide support for plugins to support various authorization mechanisms.
# Detailed information will be available in the plugin specification.
security:
  - Basic: []
  1. Set the username/password in the java code as follows:
// Configure HTTP basic authorization: Basic
    HttpBasicAuth Basic = (HttpBasicAuth) defaultClient.getAuthentication("Basic");
    Basic.setUsername("admin");
    Basic.setPassword("secret");

Upvotes: 1

Related Questions