AWS Glue: Command failed with error code 1

Question

I've been battling this error for weeks now. I have tried many different applications and cannot find any consistency with this error. Sometimes, if I change the job bookmarking setting it paused, enabled, or back to disabled it starts working. I have a java jar that I am referencing to the glue job and I am calling a few methods from it. Sometimes if I rebuild the artifact the job starts working and no longer throws this error. I have another job that uses the same exact jar and doesn't throw the error ever. I have tried creating a new job to start over, but I am seeing the same issue. Here is there error stack that it gives in the logs. The code from the application is a simple dataframe read from S3 and write to another location in s3.

val df = spark.read.parquet(source)

df.write.mode("overwrite").format("parquet").save(destination)

The error stack:

18/04/30 14:40:35 ERROR ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: com.amazonaws.services.glue.AWSGlue.getJobBookmark(Lcom/amazonaws/services/glue/model/GetJobBookmarkRequest;)Lcom/amazonaws/services/glue/model/GetJobBookmarkResult;
java.lang.NoSuchMethodError: com.amazonaws.services.glue.AWSGlue.getJobBookmark(Lcom/amazonaws/services/glue/model/GetJobBookmarkRequest;)Lcom/amazonaws/services/glue/model/GetJobBookmarkResult;
at com.amazonaws.services.glue.util.JobBookmarkServiceShim$$anonfun$2.apply(JobBookmarkUtils.scala:54)
at com.amazonaws.services.glue.util.JobBookmarkServiceShim$$anonfun$2.apply(JobBookmarkUtils.scala:54)
at scala.util.Try$.apply(Try.scala:192)
at com.amazonaws.services.glue.util.JobBookmarkServiceShim.<init>(JobBookmarkUtils.scala:54)
at com.amazonaws.services.glue.util.JobBookmark$.configure(JobBookmarkUtils.scala:178)
at com.amazonaws.services.glue.util.Job$.init(Job.scala:68)
at com.amazonaws.services.glue.util.Job$.init(Job.scala:32)
at NetezzaRawToRefined$.main(script_2018-04-30-14-39-54.scala:16)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.amazonaws.services.glue.util.GlueExceptionWrapper$$anonfun$1.apply$mcV$sp(GlueExceptionWrapper.scala:29)
at com.amazonaws.services.glue.util.GlueExceptionWrapper$$anonfun$1.apply(GlueExceptionWrapper.scala:29)
at com.amazonaws.services.glue.util.GlueExceptionWrapper$$anonfun$1.apply(GlueExceptionWrapper.scala:29)
at scala.util.Try$.apply(Try.scala:192)
at com.amazonaws.services.glue.util.GlueExceptionWrapper$.delayedEndpoint$com$amazonaws$services$glue$util$GlueExceptionWrapper$1(GlueExceptionWrapper.scala:28)
at com.amazonaws.services.glue.util.GlueExceptionWrapper$delayedInit$body.apply(GlueExceptionWrapper.scala:11)
at scala.Function0$
class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at com.amazonaws.services.glue.util.GlueExceptionWrapper$.main(GlueExceptionWrapper.scala:11)
at com.amazonaws.services.glue.util.GlueExceptionWrapper.main(GlueExceptionWrapper.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:635)
18/04/30 14:40:35 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: com.amazonaws.services.glue.AWSGlue.getJobBookmark(Lcom/amazonaws/services/glue/model/GetJobBookmarkRequest;)Lcom/amazonaws/services/glue/model/GetJobBookmarkResult;)

Answer 1

java.lang.NoSuchMethodError: com.amazonaws.services.glue.AWSGlue.getJobBookmark(Lcom/amazonaws/services/glue/model/GetJobBookmarkRequest;)Lcom/amazonaws/services/glue/model/GetJobBookmarkResult;

This error typically happens if your run time has different version of jar than your packaged version. I would suggest to make sure you are packing same version of jar.