CODEWITHSUNDEEP
amazon-web-servicesapache-sparklivy
Andrés Bernardo
Andrés Bernardo

Reputation: 62

AWS EMR Livy sessions state dead

I'm using EMR with livy, but Livy kill some sessions, is there any way to wait for other tasks to complete instead of killing those sessions? Thanks,

Here is the output for those killing sessions:

Warning: Ignoring non-spark config property: maximizeResourceAllocation=true 19/04/23 12:07:11 WARN SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.executor.memoryOverhead' instead. 19/04/23 12:07:11 WARN SparkConf: The configuration key 'spark.scheduler.listenerbus.eventqueue.size' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.scheduler.listenerbus.eventqueue.capacity' instead. 19/04/23 12:07:11 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 19/04/23 12:07:12 INFO RMProxy: Connecting to ResourceManager at ip-172-18-29-18.ec2.internal/172.18.29.18:8032 19/04/23 12:07:13 INFO Client: Requesting a new application from cluster with 4 NodeManagers 19/04/23 12:07:13 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (491520 MB per container) 19/04/23 12:07:13 INFO Client: Will allocate AM container, with 22000 MB memory including 2000 MB overhead 19/04/23 12:07:13 INFO Client: Setting up container launch context for our AM 19/04/23 12:07:13 INFO Client: Setting up the launch environment for our AM container 19/04/23 12:07:13 INFO Client: Preparing resources for our AM container 19/04/23 12:07:14 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 19/04/23 12:07:16 INFO Client: Uploading resource file:/mnt/tmp/spark-9e521492-1bdf-4115-a0d1-d3c278f7058f/spark_libs__3546840914405417444.zip -> hdfs://ip-172-18-29-18.ec2.internal:8020/user/livy/.sparkStaging/application_1556020828433_0008/__spark_libs__3546840914405417444.zip 19/04/23 12:07:17 INFO Client: Uploading resource file:/usr/lib/livy/rsc-jars/netty-all-4.0.37.Final.jar -> hdfs://ip-172-18-29-18.ec2.internal:8020/user/livy/.sparkStaging/application_1556020828433_0008/netty-all-4.0.37.Final.jar 19/04/23 12:07:17 INFO Client: Uploading resource file:/usr/lib/livy/rsc-jars/livy-rsc-0.5.0-incubating.jar -> hdfs://ip-172-18-29-18.ec2.internal:8020/user/livy/.sparkStaging/application_1556020828433_0008/livy-rsc-0.5.0-incubating.jar 19/04/23 12:07:17 INFO Client: Uploading resource file:/usr/lib/livy/rsc-jars/livy-api-0.5.0-incubating.jar -> hdfs://ip-172-18-29-18.ec2.internal:8020/user/livy/.sparkStaging/application_1556020828433_0008/livy-api-0.5.0-incubating.jar 19/04/23 12:07:17 INFO Client: Uploading resource file:/usr/lib/livy/repl_2.11-jars/livy-core_2.11-0.5.0-incubating.jar -> hdfs://ip-172-18-29-18.ec2.internal:8020/user/livy/.sparkStaging/application_1556020828433_0008/livy-core_2.11-0.5.0-incubating.jar 19/04/23 12:07:17 INFO Client: Uploading resource file:/usr/lib/livy/repl_2.11-jars/livy-repl_2.11-0.5.0-incubating.jar -> hdfs://ip-172-18-29-18.ec2.internal:8020/user/livy/.sparkStaging/application_1556020828433_0008/livy-repl_2.11-0.5.0-incubating.jar 19/04/23 12:07:17 INFO Client: Uploading resource file:/usr/lib/livy/repl_2.11-jars/commons-codec-1.9.jar -> hdfs://ip-172-18-29-18.ec2.internal:8020/user/livy/.sparkStaging/application_1556020828433_0008/commons-codec-1.9.jar 19/04/23 12:07:17 INFO Client: Uploading resource file:/etc/spark/conf/hive-site.xml -> hdfs://ip-172-18-29-18.ec2.internal:8020/user/livy/.sparkStaging/application_1556020828433_0008/hive-site.xml 19/04/23 12:07:17 INFO Client: Uploading resource file:/usr/lib/spark/R/lib/sparkr.zip#sparkr -> hdfs://ip-172-18-29-18.ec2.internal:8020/user/livy/.sparkStaging/application_1556020828433_0008/sparkr.zip 19/04/23 12:07:17 INFO Client: Uploading resource file:/usr/lib/spark/python/lib/pyspark.zip -> hdfs://ip-172-18-29-18.ec2.internal:8020/user/livy/.sparkStaging/application_1556020828433_0008/pyspark.zip 19/04/23 12:07:17 INFO Client: Uploading resource file:/usr/lib/spark/python/lib/py4j-0.10.7-src.zip -> hdfs://ip-172-18-29-18.ec2.internal:8020/user/livy/.sparkStaging/application_1556020828433_0008/py4j-0.10.7-src.zip 19/04/23 12:07:17 INFO Client: Uploading resource file:/mnt/tmp/spark-9e521492-1bdf-4115-a0d1-d3c278f7058f/__spark_conf__8433272661836653995.zip -> hdfs://ip-172-18-29-18.ec2.internal:8020/user/livy/.sparkStaging/application_1556020828433_0008/__spark_conf.zip 19/04/23 12:07:17 INFO SecurityManager: Changing view acls to: livy 19/04/23 12:07:17 INFO SecurityManager: Changing modify acls to: livy 19/04/23 12:07:17 INFO SecurityManager: Changing view acls groups to: 19/04/23 12:07:17 INFO SecurityManager: Changing modify acls groups to: 19/04/23 12:07:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(livy); groups with view permissions: Set(); users with modify permissions: Set(livy); groups with modify permissions: Set() 19/04/23 12:07:17 INFO Client: Submitting application application_1556020828433_0008 to ResourceManager 19/04/23 12:07:17 INFO YarnClientImpl: Submitted application application_1556020828433_0008 19/04/23 12:07:17 INFO Client: Application report for application_1556020828433_0008 (state: ACCEPTED) 19/04/23 12:07:17 INFO Client: client token: N/A diagnostics: [Tue Apr 23 12:07:17 +0000 2019] Application is Activated, waiting for resources to be assigned for AM. Details : AM Partition = ; Partition Resource = ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 93.24219 % ; Queue's Absolute max capacity = 100.0 % ; ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1556021237796 final status: UNDEFINED tracking URL: http://ip-172-18-29-18.ec2.internal:20888/proxy/application_1556020828433_0008/ user: livy 19/04/23 12:07:17 INFO ShutdownHookManager: Shutdown hook called 19/04/23 12:07:17 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-a5a3a313-9040-4c1c-b2d2-f51fd429064b 19/04/23 12:07:17 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-9e521492-1bdf-4115-a0d1-d3c278f7058f

Blockquote

YARN Diagnostics Application application_1556020828433_0008 was killed by user livy at 172.18.29.18

Upvotes: 1

Views: 2082

Answers (1)

dassum
dassum

Reputation: 5093

Inactive sessions will be killed after livy.server.session.timeout value expires.

use the below properties to set in livy.conf to configure session timeout

# Enabled to check whether timeout Livy sessions should be stopped.
# livy.server.session.timeout-check = true

# Time in milliseconds on how long Livy will wait before timing out an idle session.
# livy.server.session.timeout = 1h
#
# How long a finished session state should be kept in LivyServer for query.
# livy.server.session.state-retain.sec = 600s

Upvotes: 0

Related Questions