CODEWITHSUNDEEP
scalasbtapache-flink
David Magalhães
David Magalhães

Reputation: 822

"Class org.apache.hadoop.fs.s3a.S3AFileSystem not found" on fatjar with sbt assembly

I'm building a job in Scala to run on a Flink Cluster, that will store data in AWS S3, and I've some problems related to dependencies.

I've checked most of the question previously asked here, and to fix this I needed to add flink-s3-fs-hadoop-1.9.1.jar jar files to the $FLINK_HOME/plugins in order to run my job successfully:

My question is, should this be detected as been inside the fatjar generate by sbt assembly ? The files are inside the jar, but for some reason the Flink Cluster can't see them.

I know that in the documentation says that flink-s3-fs-hadoop-1.9.1.jar should download to $FLINK_HOME/plugins folder.

Upvotes: 0

Views: 807

Answers (1)

Chesnay Schepler
Chesnay Schepler

Reputation: 1270

Filesystems cannot be bundled in the user-jar, they must be present either in /lib or /plugins. The components that use filesystems aren't necessarily aware of the user-jar.

Upvotes: 1

Related Questions