CODEWITHSUNDEEP
javascalahadoopjarword-count
Piyush Jiwane
Piyush Jiwane

Reputation: 179

Exception in thread "main" java.io.IOException: Job failed

I am new to the Hadoop Environment, Just tried a WordCount program, but every time getting the error like

3732814800_0004 failed 2 times due to AM Container for appattempt_1623732814800_0004_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2021-06-15 10:45:16.241]Exception from container-launch. Container id: container_1623732814800_0004_02_000001 Exit code: 1 Exception message: CreateSymbolicLink error (1314): A required privilege is not held by the client.

Shell output: 1 file(s) moved. "Setting up env variables" "Setting up job resources"

[2021-06-15 10:45:16.244]Container exited with a non-zero exit code 1. [2021-06-15 10:45:16.245]Container exited with a non-zero exit code 1. For more detailed output, check the application tracking page: http://DESKTOP-22LEODT:8088/cluster/app/application_1623732814800_0004 Then click on links to logs of each attempt. . Failing the application. 2021-06-15 10:45:17,267 INFO mapreduce.Job: Counters: 0 Exception in thread "main" java.io.IOException: Job failed!

So what I did till now I just created a Scala class for Hadoop word count program and create a jar file from that

package piyush.jiwane.hadoop

import java.io.IOException
import java.util._
import scala.collection.JavaConversions._
import org.apache.hadoop.fs.Path
import org.apache.hadoop.conf._
import org.apache.hadoop.io._
import org.apache.hadoop.mapred._
import org.apache.hadoop.util._

object WordCount {
  class Map extends MapReduceBase with Mapper[LongWritable, Text, Text, IntWritable] {
    private final val one = new IntWritable(1)
    private val word = new Text()

    @throws[IOException]
    def map(key: LongWritable, value: Text, output: OutputCollector[Text, IntWritable], reporter: Reporter) {
      val line: String = value.toString
      line.split(" ").foreach { token =>
        word.set(token)
        output.collect(word, one)
      }
    }
  }

  class Reduce extends MapReduceBase with Reducer[Text, IntWritable, Text, IntWritable] {
    @throws[IOException]
    def reduce(key: Text, values: Iterator[IntWritable], output: OutputCollector[Text, IntWritable], reporter: Reporter) {
      val sum = values.toList.reduce((valueOne, valueTwo) => new IntWritable(valueOne.get() + valueTwo.get()))
      output.collect(key, new IntWritable(sum.get()))
    }
  }

  @throws[Exception]
  def main(args: Array[String]) {
    val conf: JobConf = new JobConf(this.getClass)
    conf.setJobName("WordCountScala")
    conf.setOutputKeyClass(classOf[Text])
    conf.setOutputValueClass(classOf[IntWritable])
    conf.setMapperClass(classOf[Map])
    conf.setCombinerClass(classOf[Reduce])
    conf.setReducerClass(classOf[Reduce])
    conf.setInputFormat(classOf[TextInputFormat])
    conf.setOutputFormat(classOf[TextOutputFormat[Text, IntWritable]])
    FileInputFormat.setInputPaths(conf, new Path(args(1)))
    FileOutputFormat.setOutputPath(conf, new Path(args(2)))
    JobClient.runJob(conf)
  }
}

the input file for hadoop program

11 23 45 17 23 45 88 15 24 26 85 96 44 52 10 15 55 84 58 62 78 98 84

For the reference i have follow the below links

https://blog.knoldus.com/hadoop-word-count-program-in-scala/ =====&&=== https://www.logicplay.club/how-to-run-hadoop-wordcount-mapreduce-example-on-windows-10/ =====&&====== org.apache.hadoop.mapred.FileAlreadyExistsException

please help to undestand this thanks in advance

EDIT after running the program in cmd administrative mode, i got the output like this

2021-06-15 11:42:46,421 INFO mapreduce.Job:  map 0% reduce 0% 2021-06-15 11:43:01,282 INFO mapreduce.Job: Task Id : attempt_1623737359186_0001_m_000000_0, Status : FAILED Error: java.lang.ClassNotFoundException: scala.collection.mutable.ArrayOps$ofRef
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        at piyush.jiwane.hadoop.WordCount$Map.map(WordCount.scala:20)
        at piyush.jiwane.hadoop.WordCount$Map.map(WordCount.scala:13)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2021-06-15 11:43:02,430 INFO mapreduce.Job: map 50% reduce 0% 2021-06-15 11:43:13,872 INFO mapreduce.Job: Task Id : attempt_1623737359186_0001_m_000000_1, Status : FAILED Error: java.lang.ClassNotFoundException: scala.collection.mutable.ArrayOps$ofRef
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        at piyush.jiwane.hadoop.WordCount$Map.map(WordCount.scala:20)
        at piyush.jiwane.hadoop.WordCount$Map.map(WordCount.scala:13)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) 2021-06-15 11:43:27,532 INFO mapreduce.Job:  map 50% reduce 17% 2021-06-15 11:43:27,562 INFO mapreduce.Job: Task Id : attempt_1623737359186_0001_m_000000_2, Status : FAILED Error: java.lang.ClassNotFoundException: scala.collection.mutable.ArrayOps$ofRef
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        at piyush.jiwane.hadoop.WordCount$Map.map(WordCount.scala:20)
        at piyush.jiwane.hadoop.WordCount$Map.map(WordCount.scala:13)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2021-06-15 11:43:28,600 INFO mapreduce.Job:  map 100% reduce 0% 2021-06-15 11:43:29,628 INFO mapreduce.Job:  map 100% reduce 100% 2021-06-15 11:43:30,669 INFO mapreduce.Job: Job job_1623737359186_0001 failed with state FAILED due to: Task failed task_1623737359186_0001_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

2021-06-15 11:43:30,877 INFO mapreduce.Job: Counters: 43
        File System Counters
                FILE: Number of bytes read=0
                FILE: Number of bytes written=236028
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=130
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=3
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=0
                HDFS: Number of bytes read erasure-coded=0
        Job Counters
                Failed map tasks=4
                Killed reduce tasks=1
                Launched map tasks=5
                Launched reduce tasks=1
                Other local map tasks=3
                Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=64773
                Total time spent by all reduces in occupied slots (ms)=24081
                Total time spent by all map tasks (ms)=64773
                Total time spent by all reduce tasks (ms)=24081
                Total vcore-milliseconds taken by all map tasks=64773
                Total vcore-milliseconds taken by all reduce tasks=24081
                Total megabyte-milliseconds taken by all map tasks=66327552
                Total megabyte-milliseconds taken by all reduce tasks=24658944
        Map-Reduce Framework
                Map input records=0
                Map output records=0
                Map output bytes=0
                Map output materialized bytes=6
                Input split bytes=96
                Combine input records=0
                Combine output records=0
                Spilled Records=0
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=19
                CPU time spent (ms)=873
                Physical memory (bytes) snapshot=250834944
                Virtual memory (bytes) snapshot=390701056 Total committed heap usage (bytes)=240123904
                Peak Map Physical memory (bytes)=250834944
                Peak Map Virtual memory (bytes)=390774784
        File Input Format Counters
                Bytes Read=34 Exception in thread "main" java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:876)
        at piyush.jiwane.hadoop.WordCount$.main(WordCount.scala:48)
        at piyush.jiwane.hadoop.WordCount.main(WordCount.scala)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

Upvotes: 2

Views: 1155

Answers (1)

Sheinbergon
Sheinbergon

Reputation: 3063

The most recent error "java.lang.ClassNotFoundException: scala.collection.mutable.ArrayOps$ofRef" comes from using an incompatible version of scala. You are using 2.13.x or newer, be sure to downgrade to 2.12.x or older.

Upvotes: 2

Related Questions