CODEWITHSUNDEEP
hadoopapache-pigubuntu-14.04
Anonymous Person
Anonymous Person

Reputation: 1538

PIG command execution

I am learning Hadoop by myself so I am not sure if what I asking is even a problem. When I run the command pig -x local to run it locally, i get the following message:


    15/10/05 15:23:28 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
    15/10/05 15:23:28 INFO pig.ExecTypeProvider: Picked LOCAL as the ExecType
    2015-10-05 15:23:28,830 [main] INFO  org.apache.pig.Main - Apache Pig version 0.15.0 (r1682971) compiled Jun 01 2015, 11:44:35
    2015-10-05 15:23:28,831 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/nkhl/pig_1444038808829.log
    2015-10-05 15:23:29,050 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /home/nkhl/.pigbootup not found
    2015-10-05 15:23:29,333 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
    2015-10-05 15:23:29,334 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
    2015-10-05 15:23:29,335 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
    2015-10-05 15:23:29,562 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum

It looks different on my online tutor's screen so I am a little confused.

What concerns me most is the deprecation part. Can someone help me with that please? What is it trying to say? Don't get me wrong, everything works fine. The GRUNT shell loads up, and things execute fine. I just wanted to know what that meant.

It's an Ubuntu machine.

Thanks!

Upvotes: 1

Views: 4426

Answers (2)

Jason Plurad
Jason Plurad

Reputation: 6782

You have some Hadoop-related variables set, such as HADOOP_HOME or HADOOP_PREFIX or HADOOP_CONF_DIR, which aren't needed if you are running Pig in local mode.

unset HADOOP_HOME
unset HADOOP_PREFIX
unset HADOOP_CONF_DIR

Deprecations aren't scary. They are a reminder that the code is calling on something that will eventually go away in a future version. These specific deprecations are caused by differences between Hadoop 1 vs Hadoop 2. Pig is compatible with both versions. If you happened to be using Hadoop 1.2.1 instead of 2.x, you wouldn't see the warnings. This is because Pig is checking the Hadoop 1 values first.

If you're interested in learning more, you can check out the Pig source code. https://github.com/apache/pig/blob/release-0.15.0/src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java#L219-L222

Upvotes: 1

maanoor99
maanoor99

Reputation: 141

Running pig as local is great AFAIK if you are using for some quick testing.Like displaying the sysout in UDF etc. The above warnings you can safely ignore.It is saying that some of the variables set in conf-site.xml are deprecated.

You can switch off those parameters by editing the

log4j.logger.org.apache.hadoop.conf.Configuration.deprecation

in log4j.properties file.

Upvotes: 3

Related Questions