CODEWITHSUNDEEP
apache-spark
Prabu Soundar Rajan
Prabu Soundar Rajan

Reputation: 819

Spark application with Jackson version 2.8 is incompatible with Apache Spark 1.6

I have a Spark application which uses Jackson 2.8 API and I use spark 1.6 as provided (scope) dependency in application pom.xml. When I try to deploy the Spark application in cluster mode, the Jackson older version from Spark 1.6 build is been picked causing the application to fail.

I tired supplying 2.8 Jackson jar through "--jars" option, build Uber application jar with latest Jackson dependency included and userClasspathFirst option on executor/driver - None of these options helped.

I placed the latest Jackson jar in all Spark worker nodes in the same location and added the path to the executor classpath option - Only in this option, the latest Jackson version is picked. In this solution every time I add a new worker node to my application, I have to place the latest Jackson which I find as a disadvantage. If someone has a better solution, please let me know.

Upvotes: 1

Views: 348

Answers (1)

Assaf Mendelson
Assaf Mendelson

Reputation: 12991

You can try to shade Jackson. For example in maven you would do something like this:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <shadedArtifactAttached>true</shadedArtifactAttached>
                        <relocations>
                            <relocation>
                                <pattern>com.fastxml.jackson</pattern>
                                <shadedPattern>com.mycompany.shaded.com.fastxml.jackson</shadedPattern>
                            </relocation>
                        </relocations>
                        <finalName>FatJarName</finalName>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

The idea is that it will basically rename the jackson package and will change your internal access to use it. Then submit the new fat jar.

Note: This does not always work (specifically if you use reflection to access jackson it can point to the wrong version).

Upvotes: 3

Related Questions