CODEWITHSUNDEEP
javahadoopoozie
Asanka Herath
Asanka Herath

Reputation: 1611

Oozie keep adding a old version of httpcore jar to classpath

I'm keep getting an exception because Oozie add a wrong version of httpcore jar to classpath. I tryed different options such as

oozie.launcher.mapreduce.task.classpath.user.precedence
oozie.launcher.mapreduce.user.classpath.first

oozie.launcher.mapreduce.task.classpath.user.precedence does not respond at all and when I use oozie.launcher.mapreduce.user.classpath.first, application cannot load even one class.

In class path I can see two versions of http-core.

httpcore-4.4.1.jar
httpcore-4.2.4.jar

When application runs in stand alone mode, I'm not getting that exception.

Exception:

    Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, java.lang.NoSuchFieldError: INSTANCE
org.apache.oozie.action.hadoop.JavaMainException: java.lang.NoSuchFieldError: INSTANCE
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
    at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.NoSuchFieldError: INSTANCE
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.<clinit>(SSLConnectionSocketFactory.java:144)
    at microsoft.exchange.webservices.data.core.ExchangeServiceBase.createConnectionSocketFactoryRegistry(ExchangeServiceBase.java:244)
    at microsoft.exchange.webservices.data.core.ExchangeServiceBase.initializeHttpClient(ExchangeServiceBase.java:198)
    at microsoft.exchange.webservices.data.core.ExchangeServiceBase.<init>(ExchangeServiceBase.java:174)
    at microsoft.exchange.webservices.data.core.ExchangeServiceBase.<init>(ExchangeServiceBase.java:179)
    at microsoft.exchange.webservices.data.core.ExchangeService.<init>(ExchangeService.java:3729)
    at com.sonasoft.sonacloud.email.dispatcher.conn.EwsConnection.getConnection(EwsConnection.java:16)
    at com.sonasoft.sonacloud.email.dispatcher.conn.EwsConnection.getConnection(EwsConnection.java:10)
    at com.sonasoft.sonacloud.email.dispatcher.utils.EwsOperations.<init>(EwsOperations.java:47)
    at com.sonasoft.sonacloud.email.dispatcher.utils.EwsOperations.getInstance(EwsOperations.java:53)
    at com.sonasoft.sonacloud.email.dispatcher.main.MainClass.main(MainClass.java:41)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
    ... 15 more

Oozie client build version: 4.2.0.2.3.2.0-2950

Any help is appreciated.

Upvotes: 5

Views: 2114

Answers (2)

Samson Scharfrichter
Samson Scharfrichter

Reputation: 9067

We have had this nasty issue with HortonWorks distro 2.3.2 (shame on them):

  • the Oozie "launcher" job always gets httpcore and httpclient in the CLASSPATH as part of the Hadoop client
  • the Oozie "launcher" job always gets httpcore and httpclient as bundled in the "Oozie" ShareLib
  • the Hive/Hive2 Sharelibs contain httpcore and httpclient in a more recent version
  • from Hadoop point of view, user.classpath.first applies to both ShareLibs so it's a 50/50 chance of getting the right order for each JAR (so a 25/75 chance overall)

Bottom line: we had to

  1. remove httpcore and httpclient from the "Oozie" ShareLib HDFS dir (duh!)
  2. raise oozie.launcher.mapreduce.job.user.classpath.first flag for all actions relying on Hive JARS (i.e. Hive action, Hive2 action, Shell action calling the JDBC driver somehow, etc.)

Post-scriptum -- the Oozie server keeps in memory a list of the JARs in each ShareLib, so that removing a JAR while the server is running will trigger errors in new jobs. If you don't want to stop the Oozie server, then the "proper way" to update a live ShareLib is to (a) create a new version in a new, time-stamped directory [check the documentation...] and (b) tell the server to resync on the newer libs with oozie admin -sharelibupdate

Upvotes: 3

Tim Biegeleisen
Tim Biegeleisen

Reputation: 521093

You want to build with your local version of the httpcore JAR, but you don't want it present in your classpath, because Hadoop will provide its own version. Then you should be using the provided scope for the httpcore JAR:

<project>
    ...
    <dependencies>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <scope>provided</scope>           <!-- this line is important -->
            <version>4.4.1</version>
        </dependency>
    </dependencies>
</project>

From the Maven documentation for provided:

This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime.

Upvotes: 0

Related Questions