CODEWITHSUNDEEP
apache-nifi
vishwanatha handadi
vishwanatha handadi

Reputation: 37

How to have load balancer for nifi handlehttprequest processor

I have 3 node Nifi cluster running(1.5 version). I have built the pipeline with handlehttprequest and intermediate set of processor for transformation/cleaning the incoming data followed by handlehttpresponse. I have provided the http end point to client applications. There are various client applications which would be posting the data through the rest end point provided by nifi. I had issue with Handlehttprequest running with execution mode as all nodes. Nifi UI becomes unresponsive after some time and I could see below error message in the nifi-app.log and I could see error message in the processor(handlehttprequest).

Changing the execution to primary node, solves the issue.

So need help with below: Handlehttprequest can't work on multinode cluster environment? If so, how to make use of multiple nodes to handle high throughput of incoming data posted by client applications? how to put load balancer for handlehttprequest endpoint and distribute the incoming data to multiple nodes on the cluster?

error message at processor level:

HandleHttpRequest[id=74ee1128-2de6-3979-a818-2b598186f7aa]       HandleHttpRequest[id=74ee1128-2de6-3979-a818-2b598186f7aa] failed to process due to org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server; rolling back session: Failed to initialize the server

==========================

Nifi-app.log

2018-08-03 14:45:22,551 INFO [Timer-Driven Process Thread-6] org.eclipse.jetty.server.Server jetty-9.4.3.v20170317
2018-08-03 14:45:22,551 ERROR [Timer-Driven Process Thread-3] o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=74ee1128-2de6-3979-a818-2b598186f7aa] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server: {}
org.apache.nifi.processor.exception.ProcessException: Failed to initialize the server
        at org.apache.nifi.processors.standard.HandleHttpRequest.onTrigger(HandleHttpRequest.java:488)
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.BindException: Cannot assign requested address
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)

==============================================

Any help in providing the direction is appreciated.

Thanks, Vish

Upvotes: 1

Views: 1581

Answers (1)

Bryan Bende
Bryan Bende

Reputation: 18630

This error generally means there is an issue with the server binding to the host or port. Since it sounds like the port is available it is probably an issue with the hostname.

In a cluster you would have to leave the hostname property blank in HandleHttpRequest so it will bind to the address of each node, or you could use a dynamic expression like ${hostname}.

For load balancing, it should be the same as any other configuration to put a load balancer in front of a web application. There are some articles that cover it already:

https://pierrevillard.com/2017/02/10/haproxy-load-balancing-in-front-of-apache-nifi/

Upvotes: 3

Related Questions