CODEWITHSUNDEEP
javahadooplogginglog4j
fuggy_yama
fuggy_yama

Reputation: 617

hadoop override log4j.properties per job

Is it possible to override log4j properties per job?
I have a multi-step program (a pipeline) that runs some hadoop jobs
but before and after it also performs some other steps.

if I run my program using java command (java -jar my_program.jar) then it runs OK but I get warning:
Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

If I run my program using hadoop command (hadoop jar my_program.jar) then default hadoop log4j.properties are loaded

In my program I log information about steps execution to a specific file. I do not need logs from hadoop map/reduce tasks. I need logs from my program because it performs many other tasks.

Should I worry about that warning?
How to override log4j properties per job to keep logs from my pipeline with my custom log4j.properties?

I am using hadoop v1.0.4

EDIT:
I discovered that after exporting LD_LIBRARY_PATH the warning Unable to load native-hadoop library for your platform... no longer appears when running my program with command java -jar my_program.jar. Source description:
http://balanceandbreath.blogspot.ca/2013/01/utilnativecodeloader-unable-to-load.html
Does someone know how to How to override log4j properties per job?

Upvotes: 3

Views: 1172

Answers (1)

fuggy_yama
fuggy_yama

Reputation: 617

Basing on this answer: https://stackoverflow.com/a/7390644/4496488
I am using folowing code to load my custom log4j.properties in the main code that launches hadoop jobs (in this case it actually overwrites the hadoop log4j config):

String log4jPropertiesFilePath = "path/to/my/custom/log4j.properties";
PropertyConfigurator.configure(log4jPropertiesFilePath);

It does not overwrite the log4j properties in datanodes - only in the program launching hadoop jobs - it is exactly what I wanted.

Upvotes: 2

Related Questions