fail to upload jar in Flink 1.5 using java rest client

Question

we got error response {"errors":["Internal server error: null"]} when uploading jar to Flink 1.5.0 server using java REST client. The same code work properly in Flink 1.4.2. In fact we can see the jar has been uploaded from Flink GUI. But the wrong status break logic. Any advice please?

        HttpPost uploadFile = new HttpPost(flinkJobManagerUrl + "/jars/upload");
        MultipartEntityBuilder builder = MultipartEntityBuilder.create();
        builder.addBinaryBody(
                "jarfile",
                new FileInputStream(f),
                ContentType.create("application/x-java-archive"),
                f.getName()
        );

        HttpEntity multipart = builder.build();
        uploadFile.setEntity(multipart);
        CloseableHttpResponse response = restClient.execute(uploadFile);
        rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));

==========================

Error from JM log:

018-06-01 16:05:46,514 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Unhandled exception org.apache.flink.shaded.netty4.io.netty.handler.codec.http.multipart.HttpPostRequestDecoder$EndOfDataDecoderException at org.apache.flink.shaded.netty4.io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.hasNext(HttpPostMultipartRequestDecoder.java:366) at org.apache.flink.shaded.netty4.io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.hasNext(HttpPostRequestDecoder.java:241) at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:92) at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:51) at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:748)

---

But after awhile you will see the jar uploaded from GUI. But then when run it it will give another exception:

2018-06-01 16:10:06,752 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler - Exception occurred in REST handler. org.apache.flink.runtime.rest.handler.RestHandlerException: Expected only one value [--KAFKA_IN a_O@192.168.56.120:9092, 192.168.56.121:9092, 192.168.56.122:9092/a_O_124 --ZK_SESSION_TIMEOUT 60000 --ZK_KEEP_CONN_ALIVE 1]. at org.apache.flink.runtime.rest.handler.util.HandlerRequestUtils.getQueryParameter(HandlerRequestUtils.java:56) at org.apache.flink.runtime.rest.handler.util.HandlerRequestUtils.getQueryParameter(HandlerRequestUtils.java:44) at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.handleRequest(JarRunHandler.java:102) at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.handleRequest(JarRunHandler.java:68) at org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:77) at org.apache.flink.runtime.rest.AbstractHandler.respondAsLeader(AbstractHandler.java:168) at org.apache.flink.runtime.rest.handler.RedirectHandler.lambda$null$0(RedirectHandler.java:139) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:748)

Answer 1

about the second exception, you can put your programArgs with comma into the post request body like

{
  "programArgs":"--test=a,b"
}

Answer 2

The first exception is due to a bug in the FileUploadHandler; it doesn't properly handle EmptyLastHttpContent messages. See https://issues.apache.org/jira/browse/FLINK-9500.

The second exception is caused by having spaces within the program-arguments, which as of 1.5 is not supported anymore.