CODEWITHSUNDEEP
shellhadoop
yanachen
yanachen

Reputation: 3753

What are the parameters' usage here in the shell code?

hadoop jar cc-jar-with-dependencies.jar com.coupang.pz.cc.merge.Merge_Run \
    ${IDF_OUT}\
    ${IG_OUT}\
    ${PROB_OUT}\
    ${MERGE_OUT}\
    1.00 \
    0.000001 \
    0.0001 \

There is a piece of shell code and I know the hadoop will run the cc-jar-with-dependencies.jar on hdfs. But what are the meaning of the other parameters below from the second line. Are they the parameters needed for the jar package ? ${...} is the path on hdfs, like ${IDF_OUT} and so on.

Upvotes: 1

Views: 42

Answers (2)

Fred
Fred

Reputation: 7005

Those parameters are passed to the hadoop command, so you would need to read the documentation for that command.

However, it might be interesting for you to find out the values contained in these parameters when your script is run. You can do that my modifying the code as shown below :

echo >&2 \
hadoop jar cc-jar-with-dependencies.jar com.coupang.pz.cc.merge.Merge_Run \
    ${IDF_OUT}\
    ${IG_OUT}\
    ${PROB_OUT}\
    ${MERGE_OUT}\
    1.00 \
    0.000001 \
    0.0001 \

This change will cause the whole command to be printed rather than executed, while the >&2 causes standard output to be output to standard error (which may help getting the data printed to the terminal if there is some output capture going on). Please note that this change is for debugging/curiosity only, it will make your script omit execution of the command.

If you know the values, the whole command is likely be easier to make sense of.

Upvotes: 1

Inian
Inian

Reputation: 85800

The usage of {WORD} is the basic case of Paramter Expansion in bash, shell

$PARAMETER

${PARAMETER}

The easiest form is to just use a parameter's name within braces. This is identical to using $FOO like you see it everywhere, but has the advantage that it can be immediately followed by characters that would be interpreted as part of the parameter name otherwise.

with an example,

word="car"
echo "The plural of $word is most likely $words"
echo "The plural of $word is most likely ${word}s"

produces an output as,

The plural of car is most likely
The plural of car is most likely cars

See the first line not containing cars as expected because shell was able to interpret only ${word} and not $words.

Coming back to your example,

hadoop jar cc-jar-with-dependencies.jar com.coupang.pz.cc.merge.Merge_Run \
    ${IDF_OUT}\
    ${IG_OUT}\
    ${PROB_OUT}\
    ${MERGE_OUT}\
    1.00 \
    0.000001 \
    0.0001 \

From the second line on-wards, the variables ${IDF_OUT}, ${IG_OUT}, ${PROB_OUT} and ${MERGE_OUT} are all in likelihood some variables (could be environment variables in the hadoop file system) which will get expanded to values when the command is run.

Whilst I have explained what the ${WORD} syntaxes are, the actual purposes of the above variables are not quite relevant in the context of shell.

Upvotes: 1

Related Questions