CODEWITHSUNDEEP
restapache-sparkcurlhadoop-yarnresourcemanager
user1326784
user1326784

Reputation: 657

How to Kill a Spark Application using Yarn ResourceManager REST API

I am trying to use Yarn REST Resource Manager API to kill a spark application running on Yarn. Below are two different PUT commands i tried to kill the application:

  1. First Command
curl -X PUT 'http://<HOSTNAME>:8088/ws/v1/cluster/apps/<APPLICATION_ID>/state' -d '{"state": "KILLED"}'

Result:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><RemoteException><exception>WebApplicationException</exception><javaClassName>javax.ws.rs.WebApplicationException</javaClassName></RemoteException>
  1. Second Command
curl -v -X PUT -H "Content-Type: application/json" -d '{"state": "KILLED"}' 'http://<HOSTNAME>:8088/ws/v1/cluster/apps/<APPLICATION_ID>/state'

Result:

* About to connect() to <HOSTNAME> port 8088 (#0)
*   Trying <IP>...
* Connected to <HOSTNAME> (<IP>) port 8088 (#0)
> PUT /ws/v1/cluster/apps/<APPLICATION_ID>/state HTTP/1.1
> User-Agent: curl/<SOME IP>
> Host: <HOSTNAME>:8088
> Accept: */*
> Content-Type: application/json
> Content-Length: 19
>
* upload completely sent off: 19 out of 19 bytes
< HTTP/1.1 403 Forbidden
< Cache-Control: no-cache
< Expires: Mon, 07 Sep 2020 18:26:46 GMT
< Date: Mon, 07 Sep 2020 18:26:46 GMT
< Pragma: no-cache
< Expires: Mon, 07 Sep 2020 18:26:46 GMT
< Date: Mon, 07 Sep 18:26:46 GMT
< Pragma: no-cache
< Content-Type: application/json
< X-Frame-Options: SAMEORIGIN
< Transfer-Encoding: chunked
< Server: Jetty(<SOME IP>.hwx)
<
* Connection #0 to host <HOSTNAME> left intact
{"RemoteException":{"exception":"ForbiddenException","message":"java.lang.Exception: The default static user cannot carry out this operation.","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"}}

Am i missing something here or do i need to provide the userID. What is the correct command to kill the application. Please suggest.

Thanks

Upvotes: 1

Views: 2001

Answers (2)

Jatin Chauhan
Jatin Chauhan

Reputation: 325

If this helps...

yarn application -kill <application_id>

Upvotes: -1

ShounenG
ShounenG

Reputation: 81

According to this ResourceManager API document, the PUT request need to be authenticated.

Generally speaking, if we mention authentication in Hadoop, the most basic is Kerberos authentication.

So you need to first confirm the Kerberos authentication of Web Console is enabled for HDFS and YARN.
You can refer to this document, if you are using Cloudera Manager to manage your CDH/CDP cluster. If you are using original Hadoop or other Hadoop product, please find corresponding document to do so.

After enabling basic authentication for cluster and for web console, you can use any way which is able to integrate with Kerberos to perform the HTTP API request.
Here is an example:

  1. Submit a MapReduce Job on c4669-node2:
[root@c4669-node2 63-hdfs-DATANODE]# yarn jar /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6626826/jars/hadoop-mapreduce-client-jobclient-3.0.0-cdh6.3.4-tests.jar sleep -Dmapred.job.queue.name=a1 -m 1 -r 1 -rt 1200000 -mt 20
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
...
21/01/08 07:06:08 INFO client.RMProxy: Connecting to ResourceManager at c4669-node4.coelab.cloudera.com/172.25.39.199:8032
21/01/08 07:06:08 INFO hdfs.DFSClient: Created token for cloudera: HDFS_DELEGATION_TOKEN [email protected], renewer=yarn, realUser=, issueDate=1610089568852, maxDate=1610694368852, sequenceNumber=4, masterKeyId=4 on 172.25.34.78:8020
21/01/08 07:06:08 INFO security.TokenCache: Got dt for hdfs://c4669-node2.coelab.cloudera.com:8020; Kind: HDFS_DELEGATION_TOKEN, Service: 172.25.34.78:8020, Ident: (token for cloudera: HDFS_DELEGATION_TOKEN [email protected], renewer=yarn, realUser=, issueDate=1610089568852, maxDate=1610694368852, sequenceNumber=4, masterKeyId=4)
21/01/08 07:06:08 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/cloudera/.staging/job_1610089441463_0001
...
21/01/08 07:06:10 INFO impl.YarnClientImpl: Submitted application application_1610089441463_0001
21/01/08 07:06:10 INFO mapreduce.Job: The url to track the job: http://c4669-node4.coelab.cloudera.com:8088/proxy/application_1610089441463_0001/
21/01/08 07:06:10 INFO mapreduce.Job: Running job: job_1610089441463_0001
21/01/08 07:06:20 INFO mapreduce.Job: Job job_1610089441463_0001 running in uber mode : false
21/01/08 07:06:20 INFO mapreduce.Job:  map 0% reduce 0%
21/01/08 07:06:25 INFO mapreduce.Job:  map 100% reduce 0%
21/01/08 07:06:42 INFO mapreduce.Job:  map 100% reduce 67%
21/01/08 07:07:06 INFO mapreduce.Job:  map 100% reduce 68%
21/01/08 07:07:11 INFO mapreduce.Job:  map 0% reduce 0%
21/01/08 07:07:11 INFO mapreduce.Job: Job job_1610089441463_0001 failed with state KILLED due to: Application application_1610089441463_0001 was killed by user cloudera
21/01/08 07:07:11 INFO mapreduce.Job: Counters: 0
[root@c4669-node2 63-hdfs-DATANODE]#

Notice:

application_1610089441463_0001 was killed by user cloudera"

is due to the PUT request from below.

  1. On c4669-node3, use curl tool to send a PUT request：
[root@c4669-node3 yum.repos.d]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: [email protected]

Valid starting       Expires              Service principal
01/08/2021 06:39:56  01/09/2021 06:39:56  krbtgt/[email protected]
01/08/2021 06:56:13  01/09/2021 06:39:56  HTTP/c4669-node4.coelab.cloudera.com@
01/08/2021 06:56:13  01/09/2021 06:39:56  HTTP/[email protected]
[root@c4669-node3 yum.repos.d]# yarn application -list -appStates 'NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUNNING'
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
21/01/08 07:00:36 INFO client.RMProxy: Connecting to ResourceManager at c4669-node4.coelab.cloudera.com/172.25.39.199:8032
Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING] and tags: []):1
                Application-Id      Application-Name        Application-Type          User           Queue                   State             Final-State             Progress                        Tracking-URL
application_1610088875054_0001             Sleep job               MAPREDUCE      cloudera         root.a1                 RUNNING               UNDEFINED               83.52% http://c4669-node4.coelab.cloudera.com:44759
[root@c4669-node3 yum.repos.d]# clear
[root@c4669-node3 yum.repos.d]# yarn application -list -appStates 'NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUNNING'
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
21/01/08 07:06:32 INFO client.RMProxy: Connecting to ResourceManager at c4669-node4.coelab.cloudera.com/172.25.39.199:8032
Total number of applications (application-types: [], states: [NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING] and tags: []):1
                Application-Id      Application-Name        Application-Type          User           Queue                   State             Final-State             Progress                        Tracking-URL
application_1610089441463_0001             Sleep job               MAPREDUCE      cloudera         root.a1                 RUNNING               UNDEFINED                  50% http://c4669-node3.coelab.cloudera.com:35559
[root@c4669-node3 yum.repos.d]# curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt http://c4669-node4.coelab.cloudera.com:8088/ws/v1/cluster/apps/application_1610089441463_0001/state
{"state":"RUNNING"}[root@c4669-node3 yum.repos.d]# curl --negotiate -u : -b ~/cookiejar.txt -c ~/cookiejar.txt -XPUT -H "Content-type: application/json" -d '{
>   "state":"KILLED"
> }' 'http://c4669-node4.coelab.cloudera.com:8088/ws/v1/cluster/apps/application_1610089441463_0001/state'
{"state":"FINAL_SAVING"}[root@c4669-node3 yum.repos.d]#

Upvotes: 0

Related Questions