CODEWITHSUNDEEP
pythonlinuxdockerapache-sparktcp
KAI
KAI

Reputation: 43

Local spark docker cannot connect to remote spark master docker

I'm new to spark, so if there is any naive or duplicated problem, I apologize in advance.

There are four bitnami dockers. One is build on local machine, and the resting are build on remote virtual machines with host 11.119.9.1,11.119.9.2,11.119.9.3 respectively. One of them with host 11.119.9.1 and port 7077 is spark-master node. The local machine connect to remote one through VPN.

What I'd like to do is to submit local pyspark script in the docker in local machine to the virtual machine of spark master in remote server with cluster mode. After executing spark-sumbit, I checked log of master node of Spark web UI and found the error "java.io.IOException: Failed to connect to spark-master/11.119.8.1:35369" below. It seems that the worker failed to connect to the master.

Spark Executor Command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" 
"-Xmx1024M" "-Dspark.driver.port=35369" "-Dspark.cassandra.connection.port=9042" 
"-Djava.net.preferIPv6Addresses=false" "-XX:+IgnoreUnrecognizedVMOptions" 
"--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" 
"--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" 
"--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" 
"--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" 
"--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" 
"--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" 
"--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" 
"--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" 
"-Djdk.reflect.useDirectMethodHandle=false" "org.apache.spark.executor.CoarseGrainedExecutorBackend" 
"--driver-url" "spark://CoarseGrainedScheduler@spark-master:35369" "--executor-id" "17" "--hostname" "11.119.8.3" 
"--cores" "4" "--app-id" "app-20250221160412-0007" "--worker-url" "spark://[email protected]:40277" "--resourceProfileId" "0"


Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
25/02/21 16:04:26 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 51665@s-datacenter03
25/02/21 16:04:26 INFO SignalUtils: Registering signal handler for TERM
25/02/21 16:04:26 INFO SignalUtils: Registering signal handler for HUP
25/02/21 16:04:26 INFO SignalUtils: Registering signal handler for INT
25/02/21 16:04:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
25/02/21 16:04:26 INFO SecurityManager: Changing view acls to: spark
25/02/21 16:04:26 INFO SecurityManager: Changing modify acls to: spark
25/02/21 16:04:26 INFO SecurityManager: Changing view acls groups to: 
25/02/21 16:04:26 INFO SecurityManager: Changing modify acls groups to: 
25/02/21 16:04:26 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: spark; 
groups with view permissions: EMPTY; 
users with modify permissions: spark; groups with modify permissions: EMPTY
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
    at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
    at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:429)
    at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:418)
    at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult: 
    at org.apache.spark.util.SparkThreadUtils$.awaitResult(SparkThreadUtils.scala:56)
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:310)
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
    at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:102)
    at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$9(CoarseGrainedExecutorBackend.scala:449)
    at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
    at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
    at scala.collection.immutable.Range.foreach(Range.scala:158)
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
    at org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$7(CoarseGrainedExecutorBackend.scala:447)
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
    at java.base/java.security.AccessController.doPrivileged(Unknown Source)
    at java.base/javax.security.auth.Subject.doAs(Unknown Source)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
    ... 4 more
Caused by: java.io.IOException: Failed to connect to spark-master/11.119.8.1:35369
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:294)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:214)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:226)
    at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: spark-master/11.119.8.1:35369
Caused by: java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.Net.pollConnect(Native Method)
    at java.base/sun.nio.ch.Net.pollConnectNow(Unknown Source)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Unknown Source)

Of top of my head, I though it might be network issue, so I check the network connection between remote master vm and my local docker. It shows the network is connected.

root@spark-master:/opt/spark# telnet 11.119.8.1 7077
Trying 11.119.8.1...
Connected to 11.119.8.1.
Escape character is '^]'.

Then, I check connection among master node and worker nodes. It seems connections among nodes is normal.

kaikaiwan@s-datacenter02:~$ telnet 11.119.8.1 7077
Trying 11.119.8.1...
Connected to 11.119.8.1.
Escape character is '^]'.

kaikaiwan@s-datacenter03:~$ telnet 11.119.8.1 7077
Trying 11.119.8.1...
Connected to 11.119.8.1.
Escape character is '^]'.

kaikaiwan@s-datacenter01:/$ telnet 11.119.8.2 7000
Trying 11.119.8.2...
Connected to 11.119.8.2.

Then, my college run spark-submit directly in the spark master docker in the remote server. The log shows "25/02/20 19:56:39 INFO TransportClientFactory: Successfully created connection to /11.119.8.1:34277 after 77 ms (0 ms spent in bootstraps)". This seems to successfully create connection between master and worker.

Spark Executor Command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx1024M" "-Dspark.driver.port=34277" "-Djava.net.preferIPv6Addresses=false" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "-Djdk.reflect.useDirectMethodHandle=false" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://[email protected]:34277" "--executor-id" "1" "--hostname" "11.119.8.3" "--cores" "4" "--app-id" "app-20250220195646-0000" "--worker-url" "spark://[email protected]:40277" "--resourceProfileId" "0"
========================================

Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
25/02/20 19:56:38 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 105@s-datacenter03
25/02/20 19:56:38 INFO SignalUtils: Registering signal handler for TERM
25/02/20 19:56:38 INFO SignalUtils: Registering signal handler for HUP
25/02/20 19:56:38 INFO SignalUtils: Registering signal handler for INT
25/02/20 19:56:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
25/02/20 19:56:38 INFO SecurityManager: Changing view acls to: spark
25/02/20 19:56:38 INFO SecurityManager: Changing modify acls to: spark
25/02/20 19:56:38 INFO SecurityManager: Changing view acls groups to: 
25/02/20 19:56:38 INFO SecurityManager: Changing modify acls groups to: 
25/02/20 19:56:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: spark; groups with view permissions: EMPTY; users with modify permissions: spark; groups with modify permissions: EMPTY
25/02/20 19:56:39 INFO TransportClientFactory: Successfully created connection to /11.119.8.1:34277 after 77 ms (0 ms spent in bootstraps)
25/02/20 19:56:39 INFO SecurityManager: Changing view acls to: spark
25/02/20 19:56:39 INFO SecurityManager: Changing modify acls to: spark
25/02/20 19:56:39 INFO SecurityManager: Changing view acls groups to: 
25/02/20 19:56:39 INFO SecurityManager: Changing modify acls groups to: 
25/02/20 19:56:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: spark; groups with view permissions: EMPTY; users with modify permissions: spark; groups with modify permissions: EMPTY
25/02/20 19:56:39 INFO TransportClientFactory: Successfully created connection to /11.119.8.1:34277 after 5 ms (0 ms spent in bootstraps)
25/02/20 19:56:39 INFO DiskBlockManager: Created local directory at /tmp/spark-f5a3cd2f-8c1d-461f-91e2-d89c53422774/executor-ecbd4c16-7d1f-4310-9f57-e8c0176d4411/blockmgr-db0ccc37-43db-47ab-a2c3-fdaccc28b467
25/02/20 19:56:39 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
25/02/20 19:56:39 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://[email protected]:34277
25/02/20 19:56:39 INFO WorkerWatcher: Connecting to worker spark://[email protected]:40277
25/02/20 19:56:39 INFO TransportClientFactory: Successfully created connection to /11.119.8.3:40277 after 9 ms (0 ms spent in bootstraps)
25/02/20 19:56:39 INFO WorkerWatcher: Successfully connected to spark://[email protected]:40277
25/02/20 19:56:39 INFO ResourceUtils: ==============================================================
25/02/20 19:56:39 INFO ResourceUtils: No custom resources configured for spark.executor.
25/02/20 19:56:39 INFO ResourceUtils: ==============================================================
25/02/20 19:56:39 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
25/02/20 19:56:39 INFO Executor: Starting executor ID 1 on host 11.110.9.3
25/02/20 19:56:39 INFO Executor: OS info Linux, 6.8.0-41-generic, amd64
25/02/20 19:56:39 INFO Executor: Java version 17.0.14
25/02/20 19:56:39 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37575.
25/02/20 19:56:39 INFO NettyBlockTransferService: Server created on 11.110.9.3:37575
25/02/20 19:56:39 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
25/02/20 19:56:39 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, 11.119.8.3, 37575, None)
25/02/20 19:56:39 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, 11.119.8.3, 37575, None)
25/02/20 19:56:39 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, 11.119.8.3, 37575, None)
25/02/20 19:56:39 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
25/02/20 19:56:39 INFO Executor: Created or updated repl class loader org.apache.spark.util.MutableURLClassLoader@61224013 for default.
25/02/20 19:56:39 INFO Executor: Fetching spark://11.119.8.1:34277/jars/spark-examples_2.12-3.5.4.jar with timestamp 1740052605090
25/02/20 19:56:39 INFO TransportClientFactory: Successfully created connection to /11.119.8.1:34277 after 2 ms (0 ms spent in bootstraps)
25/02/20 19:56:39 INFO Utils: Fetching spark://11.119.8.1:34277/jars/spark-examples_2.12-3.5.4.jar to /tmp/spark-f5a3cd2f-8c1d-461f-91e2-d89c53422774/executor-ecbd4c16-7d1f-4310-9f57-e8c0176d4411/spark-adf57770-ca41-47d1-9179-b536932536e7/fetchFileTemp9485593455226099991.tmp
25/02/20 19:56:39 INFO Utils: Copying /tmp/spark-f5a3cd2f-8c1d-461f-91e2-d89c53422774/executor-ecbd4c16-7d1f-4310-9f57-e8c0176d4411/spark-adf57770-ca41-47d1-9179-b536932536e7/15260078331740052605090_cache to /opt/bitnami/spark/work/app-20250220195646-0000/1/./spark-examples_2.12-3.5.4.jar
25/02/20 19:56:39 INFO Executor: Adding file:/opt/bitnami/spark/work/app-20250220195646-0000/1/./spark-examples_2.12-3.5.4.jar to class loader default
25/02/20 19:56:39 INFO CoarseGrainedExecutorBackend: Got assigned task 4
25/02/20 19:56:39 INFO CoarseGrainedExecutorBackend: Got assigned task 5
25/02/20 19:56:39 INFO CoarseGrainedExecutorBackend: Got assigned task 6
25/02/20 19:56:39 INFO CoarseGrainedExecutorBackend: Got assigned task 7
25/02/20 19:56:39 INFO Executor: Running task 6.0 in stage 0.0 (TID 6)
25/02/20 19:56:39 INFO Executor: Running task 4.0 in stage 0.0 (TID 4)
25/02/20 19:56:39 INFO Executor: Running task 5.0 in stage 0.0 (TID 5)
25/02/20 19:56:39 INFO Executor: Running task 7.0 in stage 0.0 (TID 7)
25/02/20 19:56:40 INFO TorrentBroadcast: Started reading broadcast variable 0 with 1 pieces (estimated total size 4.0 MiB)
25/02/20 19:56:40 INFO TransportClientFactory: Successfully created connection to /11.119.8.1:36735 after 3 ms (0 ms spent in bootstraps)
25/02/20 19:56:40 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.3 KiB, free 434.4 MiB)
25/02/20 19:56:40 INFO TorrentBroadcast: Reading broadcast variable 0 took 117 ms
25/02/20 19:56:40 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 4.0 KiB, free 434.4 MiB)
25/02/20 19:56:40 INFO Executor: Finished task 4.0 in stage 0.0 (TID 4). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO Executor: Finished task 7.0 in stage 0.0 (TID 7). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO Executor: Finished task 5.0 in stage 0.0 (TID 5). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO Executor: Finished task 6.0 in stage 0.0 (TID 6). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 8
25/02/20 19:56:40 INFO Executor: Running task 8.0 in stage 0.0 (TID 8)
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 9
25/02/20 19:56:40 INFO Executor: Running task 9.0 in stage 0.0 (TID 9)
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 10
25/02/20 19:56:40 INFO Executor: Running task 10.0 in stage 0.0 (TID 10)
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 11
25/02/20 19:56:40 INFO Executor: Running task 11.0 in stage 0.0 (TID 11)
25/02/20 19:56:40 INFO Executor: Finished task 8.0 in stage 0.0 (TID 8). 969 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 16
25/02/20 19:56:40 INFO Executor: Running task 16.0 in stage 0.0 (TID 16)
25/02/20 19:56:40 INFO Executor: Finished task 10.0 in stage 0.0 (TID 10). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO Executor: Finished task 9.0 in stage 0.0 (TID 9). 1012 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 19
25/02/20 19:56:40 INFO Executor: Running task 19.0 in stage 0.0 (TID 19)
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 20
25/02/20 19:56:40 INFO Executor: Running task 20.0 in stage 0.0 (TID 20)
25/02/20 19:56:40 INFO Executor: Finished task 16.0 in stage 0.0 (TID 16). 1012 bytes result sent to driver
25/02/20 19:56:40 INFO Executor: Finished task 11.0 in stage 0.0 (TID 11). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 24
25/02/20 19:56:40 INFO Executor: Running task 24.0 in stage 0.0 (TID 24)
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 26
25/02/20 19:56:40 INFO Executor: Running task 26.0 in stage 0.0 (TID 26)
25/02/20 19:56:40 INFO Executor: Finished task 19.0 in stage 0.0 (TID 19). 1012 bytes result sent to driver
25/02/20 19:56:40 INFO Executor: Finished task 20.0 in stage 0.0 (TID 20). 1012 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 28
25/02/20 19:56:40 INFO Executor: Running task 28.0 in stage 0.0 (TID 28)
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 29
25/02/20 19:56:40 INFO Executor: Running task 29.0 in stage 0.0 (TID 29)
25/02/20 19:56:40 INFO Executor: Finished task 24.0 in stage 0.0 (TID 24). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 33
25/02/20 19:56:40 INFO Executor: Running task 33.0 in stage 0.0 (TID 33)
25/02/20 19:56:40 INFO Executor: Finished task 26.0 in stage 0.0 (TID 26). 1012 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 34
25/02/20 19:56:40 INFO Executor: Running task 34.0 in stage 0.0 (TID 34)
25/02/20 19:56:40 INFO Executor: Finished task 28.0 in stage 0.0 (TID 28). 1012 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 36
25/02/20 19:56:40 INFO Executor: Running task 36.0 in stage 0.0 (TID 36)
25/02/20 19:56:40 INFO Executor: Finished task 29.0 in stage 0.0 (TID 29). 1055 bytes result sent to driver
25/02/20 19:56:40 INFO CoarseGrainedExecutorBackend: Got assigned task 37
25/02/20 19:56:40 INFO Executor: Running task 37.0 in stage 0.0 (TID 37)
25/02/20 19:56:41 INFO Executor: Finished task 33.0 in stage 0.0 (TID 33). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 34.0 in stage 0.0 (TID 34). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 41
25/02/20 19:56:41 INFO Executor: Running task 41.0 in stage 0.0 (TID 41)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 42
25/02/20 19:56:41 INFO Executor: Running task 42.0 in stage 0.0 (TID 42)
25/02/20 19:56:41 INFO Executor: Finished task 37.0 in stage 0.0 (TID 37). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 45
25/02/20 19:56:41 INFO Executor: Running task 45.0 in stage 0.0 (TID 45)
25/02/20 19:56:41 INFO Executor: Finished task 36.0 in stage 0.0 (TID 36). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 46
25/02/20 19:56:41 INFO Executor: Running task 46.0 in stage 0.0 (TID 46)
25/02/20 19:56:41 INFO Executor: Finished task 41.0 in stage 0.0 (TID 41). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 50
25/02/20 19:56:41 INFO Executor: Running task 50.0 in stage 0.0 (TID 50)
25/02/20 19:56:41 INFO Executor: Finished task 45.0 in stage 0.0 (TID 45). 1055 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 42.0 in stage 0.0 (TID 42). 1055 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 51
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 52
25/02/20 19:56:41 INFO Executor: Running task 51.0 in stage 0.0 (TID 51)
25/02/20 19:56:41 INFO Executor: Running task 52.0 in stage 0.0 (TID 52)
25/02/20 19:56:41 INFO Executor: Finished task 46.0 in stage 0.0 (TID 46). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 55
25/02/20 19:56:41 INFO Executor: Running task 55.0 in stage 0.0 (TID 55)
25/02/20 19:56:41 INFO Executor: Finished task 50.0 in stage 0.0 (TID 50). 1055 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 57
25/02/20 19:56:41 INFO Executor: Running task 57.0 in stage 0.0 (TID 57)
25/02/20 19:56:41 INFO Executor: Finished task 52.0 in stage 0.0 (TID 52). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 51.0 in stage 0.0 (TID 51). 969 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 61
25/02/20 19:56:41 INFO Executor: Running task 61.0 in stage 0.0 (TID 61)
25/02/20 19:56:41 INFO Executor: Finished task 55.0 in stage 0.0 (TID 55). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 57.0 in stage 0.0 (TID 57). 969 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 62
25/02/20 19:56:41 INFO Executor: Running task 62.0 in stage 0.0 (TID 62)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 63
25/02/20 19:56:41 INFO Executor: Running task 63.0 in stage 0.0 (TID 63)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 64
25/02/20 19:56:41 INFO Executor: Running task 64.0 in stage 0.0 (TID 64)
25/02/20 19:56:41 INFO Executor: Finished task 61.0 in stage 0.0 (TID 61). 969 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 66
25/02/20 19:56:41 INFO Executor: Running task 66.0 in stage 0.0 (TID 66)
25/02/20 19:56:41 INFO Executor: Finished task 62.0 in stage 0.0 (TID 62). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 63.0 in stage 0.0 (TID 63). 1055 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 68
25/02/20 19:56:41 INFO Executor: Running task 68.0 in stage 0.0 (TID 68)
25/02/20 19:56:41 INFO Executor: Finished task 64.0 in stage 0.0 (TID 64). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 71
25/02/20 19:56:41 INFO Executor: Running task 71.0 in stage 0.0 (TID 71)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 74
25/02/20 19:56:41 INFO Executor: Running task 74.0 in stage 0.0 (TID 74)
25/02/20 19:56:41 INFO Executor: Finished task 68.0 in stage 0.0 (TID 68). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 66.0 in stage 0.0 (TID 66). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 76
25/02/20 19:56:41 INFO Executor: Running task 76.0 in stage 0.0 (TID 76)
25/02/20 19:56:41 INFO Executor: Finished task 74.0 in stage 0.0 (TID 74). 969 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 77
25/02/20 19:56:41 INFO Executor: Running task 77.0 in stage 0.0 (TID 77)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 78
25/02/20 19:56:41 INFO Executor: Finished task 71.0 in stage 0.0 (TID 71). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Running task 78.0 in stage 0.0 (TID 78)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 79
25/02/20 19:56:41 INFO Executor: Running task 79.0 in stage 0.0 (TID 79)
25/02/20 19:56:41 INFO Executor: Finished task 76.0 in stage 0.0 (TID 76). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 78.0 in stage 0.0 (TID 78). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 79.0 in stage 0.0 (TID 79). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 77.0 in stage 0.0 (TID 77). 969 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 84
25/02/20 19:56:41 INFO Executor: Running task 84.0 in stage 0.0 (TID 84)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 85
25/02/20 19:56:41 INFO Executor: Running task 85.0 in stage 0.0 (TID 85)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 86
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 87
25/02/20 19:56:41 INFO Executor: Running task 87.0 in stage 0.0 (TID 87)
25/02/20 19:56:41 INFO Executor: Running task 86.0 in stage 0.0 (TID 86)
25/02/20 19:56:41 INFO Executor: Finished task 85.0 in stage 0.0 (TID 85). 1055 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 86.0 in stage 0.0 (TID 86). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 84.0 in stage 0.0 (TID 84). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 93
25/02/20 19:56:41 INFO Executor: Finished task 87.0 in stage 0.0 (TID 87). 1012 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Running task 93.0 in stage 0.0 (TID 93)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 94
25/02/20 19:56:41 INFO Executor: Running task 94.0 in stage 0.0 (TID 94)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 95
25/02/20 19:56:41 INFO Executor: Running task 95.0 in stage 0.0 (TID 95)
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 97
25/02/20 19:56:41 INFO Executor: Running task 97.0 in stage 0.0 (TID 97)
25/02/20 19:56:41 INFO Executor: Finished task 94.0 in stage 0.0 (TID 94). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 93.0 in stage 0.0 (TID 93). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 95.0 in stage 0.0 (TID 95). 969 bytes result sent to driver
25/02/20 19:56:41 INFO Executor: Finished task 97.0 in stage 0.0 (TID 97). 969 bytes result sent to driver
25/02/20 19:56:41 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
25/02/20 19:56:41 INFO MemoryStore: MemoryStore cleared
25/02/20 19:56:41 INFO BlockManager: BlockManager stopped
25/02/20 19:56:41 INFO ShutdownHookManager: Shutdown hook called
25/02/20 19:56:41 INFO ShutdownHookManager: Deleting directory /tmp/spark-f5a3cd2f-8c1d-461f-91e2-d89c53422774/executor-ecbd4c16-7d1f-4310-9f57-e8c0176d4411/spark-adf57770-ca41-47d1-9179-b536932536e7
25/02/20 19:56:41 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
5a3cd2f-8c1d-461f-91e2-d89c53422774/executor-ecbd4c16-7d1f-4310-9f57-e8c0176d4411/spark-adf57770-ca41-47d1-9179-b536932536e7

Could anyone give me instructions on the error "java.io.IOException: Failed to connect to spark-master/11.119.8.1:35369"?

Any assistance would be greatly appreciated.

Upvotes: 1

Views: 52

Answers (0)

Related Questions