Unable run sqoop job in oozie (No enum constant com.cloudera.sqoop.SqoopOptions.FileLayout.ParquetFile)

Question

I successfully created and execute sqoop import job, but unable to run it in oozie workflow. Sqoop imports the data from RDBMS to parquet file in HDFS. It seems the problem is related to parquet format. If I use --as-textfile, the workflow run without any problem.

Also, I've copied all parquet-*.jar from SQOOP_HOME/lib to oozie share lib.

Sqoop 1.4.7 and Oozie 4.3.1.

Sqoop job definition

$ sqoop job --create ingest_amsp_custmaster -- import --connect "jdbc:oracle:thin:@<IP>:<PORT>/<SID>" \
--username <USER> -P \
--table CUSTMASTER \
--as-parquetfile \
--target-dir /warehouse/raw/amsp/custmaster \
--delete-target-dir \
-m 1

Here's what I got from error log:

java.lang.IllegalArgumentException: No enum constant com.cloudera.sqoop.SqoopOptions.FileLayout.ParquetFile
        at java.lang.Enum.valueOf(Enum.java:238)
        at org.apache.sqoop.SqoopOptions.loadProperties(SqoopOptions.java:522)
        at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:299)
        at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:198)
        at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
        at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:200)
        at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:183)
        at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:64)
        at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:51)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:436)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:350)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:211)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:254)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Intercepting System.exit(1)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

Any help would be appreciated.

Thanks. Yusata.

Answer 1

Check version of Sqoop in Oozie workflow. It seems like it is not 1.4.7: stack trace line

org.apache.sqoop.SqoopOptions.loadProperties(SqoopOptions.java:522)

is corresponds to https://github.com/apache/sqoop/blob/20af67ef60096b17e1d9585670e5ec787eb760e2/src/java/org/apache/sqoop/SqoopOptions.java#L522