CODEWITHSUNDEEP
hadoopmapreducehadoop2
Bourne
Bourne

Reputation: 1927

Where is the classpath set for hadoop

Where is the classpath for hadoop set? When I run the below command it gives me the classpath. Where is the classpath set?

  bin/hadoop classpath

I'm using hadoop 2.6.0

Upvotes: 11

Views: 54438

Answers (6)

DieterDP
DieterDP

Reputation: 4347

If you want to add classpath entries without changes to the installed files (eg, due to permission restrictions), you can use put a .hadooprc file in your home directory with following content:

hadoop_add_classpath /path/to/file1.jar
hadoop_add_classpath /path/to/file2.jar

The .hadooprc file gets sourced automatically by hadoop-config.sh if it exists.

Note this only adds classpath entries for the current user.

Upvotes: 0

Sky
Sky

Reputation: 43

I also encountered the problem and have solved it, but my hadoop version is 2.10.1.

I hope it have some help for people who use a newer hadoop version. So far, the following methods should have worked as well in the latest hadoop version 3.3.0.

You just need to edit your .bashrc or .profile, I will give an example of .bashrc.

# edit .bashrc
$ vim ~/.bashrc

Add HADOOP_HOME, PATH of hadoop bin direcotry and HADOOP_CLASSPATH in .bashrc.

# export HADOOP_HOME=${your hadoop install directory}, an example as follows:

export HADOOP_HOME=/usr/local/hadoop-2.10.1

export PATH=${HADOOP_HOME}/bin:${PATH}

export HADOOP_CLASSPATH=`hadoop classpath`

Then,

$ source ~/.bashrc

Upvotes: 1

Siva Krishnan
Siva Krishnan

Reputation: 91

Open your bash profile (~/.profile or ~/.bash_profile) for editing and add the following:

  1. export HADOOP_HOME="/usr/local/Cellar/hadoop" then Replace with your own path

  2. export HADOOP_CLASSPATH=$(find $HADOOP_HOME -name '*.jar' | xargs echo | tr ' ' ':') Save the changes and reload.

  3. source ~/.profile

Upvotes: 9

SMA
SMA

Reputation: 37023

When you run hadoop command, it sources a file hadoop-config.sh that resides in $HADOOP_HDFS_HOME/libexec which sets your classpath (CLASSPATH) by picking jars residing in various directories viz. 

$HADOOP_HDFS_HOME/share/hadoop/mapreduce 
$HADOOP_HDFS_HOME/share/hadoop/common
$HADOOP_HDFS_HOME/share/hadoop/hdfs etc.

Upvotes: 2

Ashrith
Ashrith

Reputation: 6855

As said by almas shaikh it's set in hadoop-config.sh, but you could add more jars to it in hadoop-env.sh

Here is a relevant code from hadoop-env.sh which adds additional jars like capacity-scheduler and aws jar's.

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

# Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
  if [ "$HADOOP_CLASSPATH" ]; then
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
  else
    export HADOOP_CLASSPATH=$f
  fi
done

# ... some other lines omitted

# Add Aws jar
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:share/hadoop/tools/lib/*

Upvotes: 6

Ming
Ming

Reputation: 1693

As per this blog post, it is in an environment variable named HADOOP_CLASSPATH. You can set it as you would any other environment variable, the specifics of which depend on which shell you use. If you use bash, then you can call like export HADOOP_CLASSPATH=/path/to/wherever:/path/to/wherever/else.

Upvotes: 1

Related Questions