CODEWITHSUNDEEP
javahazelcasthazelcast-jet
user3458271
user3458271

Reputation: 660

Hazelcast adding address to blacklist and client not connecting

I am new to Hazelcast and in my application when I am starting Hazelcast member it is showing two addresses been blacklisted don't know why, what is the best configuration for Hazelcast clustering?

Here is the log that I am getting:

2021-07-22 13:15:51.823  INFO 212124 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8081 (http)
2021-07-22 13:15:51.844  INFO 212124 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2021-07-22 13:15:51.845  INFO 212124 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.44]
2021-07-22 13:15:52.094  INFO 212124 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2021-07-22 13:15:52.094  INFO 212124 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 4676 ms
2021-07-22 13:15:52.417  INFO 212124 --- [           main] com.hazelcast.instance.AddressPicker     : [LOCAL] [APP] [4.2] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [192.168.0.104]
2021-07-22 13:15:53.350  INFO 212124 --- [           main] com.hazelcast.system                     : [192.168.0.104]:5701 [APP] [4.2] Hazelcast 4.2 (20210324 - 405cfd1) starting at [192.168.0.104]:5701
2021-07-22 13:15:54.178  INFO 212124 --- [           main] com.hazelcast.instance.impl.Node         : [192.168.0.104]:5701 [APP] [4.2] Using TCP/IP discovery
2021-07-22 13:15:54.180  WARN 212124 --- [           main] com.hazelcast.cp.CPSubsystem             : [192.168.0.104]:5701 [APP] [4.2] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2021-07-22 13:15:54.680  INFO 212124 --- [           main] c.h.internal.diagnostics.Diagnostics     : [192.168.0.104]:5701 [APP] [4.2] Diagnostics started
2021-07-22 13:15:54.680  INFO 212124 --- [           main] c.h.i.diagnostics.BuildInfoPlugin        : [192.168.0.104]:5701 [APP] [4.2] Plugin:active
2021-07-22 13:15:54.681  INFO 212124 --- [           main] c.h.i.d.SystemPropertiesPlugin           : [192.168.0.104]:5701 [APP] [4.2] Plugin:active
2021-07-22 13:15:54.681  INFO 212124 --- [           main] c.h.i.d.ConfigPropertiesPlugin           : [192.168.0.104]:5701 [APP] [4.2] Plugin:active
2021-07-22 13:15:54.684  INFO 212124 --- [           main] c.h.i.d.PendingInvocationsPlugin         : [192.168.0.104]:5701 [APP] [4.2] Plugin:active: period-millis:30000 threshold:1
2021-07-22 13:15:54.686  INFO 212124 --- [           main] c.h.internal.diagnostics.MetricsPlugin   : [192.168.0.104]:5701 [APP] [4.2] Plugin:active, period-millis:60000
2021-07-22 13:15:54.686  INFO 212124 --- [           main] c.h.i.diagnostics.SlowOperationPlugin    : [192.168.0.104]:5701 [APP] [4.2] Plugin:active, period-millis:30000
2021-07-22 13:15:54.687  INFO 212124 --- [           main] c.h.i.d.PendingInvocationsPlugin         : [192.168.0.104]:5701 [APP] [4.2] Plugin:active: period-millis:30000 threshold-millis:5000
2021-07-22 13:15:54.687  INFO 212124 --- [           main] c.h.i.d.PendingInvocationsPlugin         : [192.168.0.104]:5701 [APP] [4.2] Plugin:active: period-millis:5000
2021-07-22 13:15:54.688  INFO 212124 --- [           main] c.h.i.d.OperationProfilerPlugin          : [192.168.0.104]:5701 [APP] [4.2] Plugin:active, period-millis:5000
2021-07-22 13:15:54.688  INFO 212124 --- [           main] .h.i.d.MemberHazelcastInstanceInfoPlugin : [192.168.0.104]:5701 [APP] [4.2] Plugin:active, period-millis:60000
2021-07-22 13:15:54.689  INFO 212124 --- [           main] c.h.i.diagnostics.SystemLogPlugin        : [192.168.0.104]:5701 [APP] [4.2] Plugin:active: logPartitions:false
2021-07-22 13:15:54.692  INFO 212124 --- [SchedulerThread] c.h.internal.diagnostics.Diagnostics     : [192.168.0.104]:5701 [APP] [4.2] Diagnostics log directory is [D:\Parth\Hazelcast\logs]
2021-07-22 13:15:54.694  INFO 212124 --- [           main] c.h.i.diagnostics.StoreLatencyPlugin     : [192.168.0.104]:5701 [APP] [4.2] Plugin:active: period-millis:60000 resetPeriod-millis:0
2021-07-22 13:15:54.696  INFO 212124 --- [           main] .h.i.d.MemberHazelcastInstanceInfoPlugin : [192.168.0.104]:5701 [APP] [4.2] Plugin:active, period-millis:10000
2021-07-22 13:15:54.697  INFO 212124 --- [           main] c.h.i.d.OperationHeartbeatPlugin         : [192.168.0.104]:5701 [APP] [4.2] Plugin:active: period-millis:10000 max-deviation:33%
2021-07-22 13:15:54.699  INFO 212124 --- [           main] com.hazelcast.core.LifecycleService      : [192.168.0.104]:5701 [APP] [4.2] [192.168.0.104]:5701 is STARTING
2021-07-22 13:15:59.367  INFO 212124 --- [cached.thread-3] c.h.internal.cluster.impl.TcpIpJoiner    : [192.168.0.104]:5701 [APP] [4.2] [192.168.0.104]:5702 is added to the blacklist.
2021-07-22 13:15:59.524  INFO 212124 --- [cached.thread-5] c.h.internal.cluster.impl.TcpIpJoiner    : [192.168.0.104]:5701 [APP] [4.2] [192.168.0.104]:5703 is added to the blacklist.
2021-07-22 13:16:00.178  INFO 212124 --- [           main] c.h.internal.cluster.ClusterService      : [192.168.0.104]:5701 [APP] [4.2] 

Members {size:1, ver:1} [
    Member [192.168.0.104]:5701 - c35ebca4-ca5f-404c-ba0e-1c5e1287aa21 this
]

2021-07-22 13:16:00.194  INFO 212124 --- [           main] com.hazelcast.core.LifecycleService      : [192.168.0.104]:5701 [APP] [4.2] [192.168.0.104]:5701 is STARTED
2021-07-22 13:16:00.514  INFO 212124 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2021-07-22 13:16:01.489  INFO 212124 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8081 (http) with context path ''
2021-07-22 13:16:01.506  INFO 212124 --- [           main] com.APPhub.Application                   : Started Application in 15.086 seconds (JVM running for 15.996)

Here two addresses getting blacklisted not sure why.

And my server config is as below:

        Config config = new Config();
        
        config.getNetworkConfig().setPort(5701)
        .setPortAutoIncrement( true ).setPortCount(20);
        
        config.getNetworkConfig().setReuseAddress( true );
        

        System.setProperty( "hazelcast.diagnostics.enabled", "true" );
        System.setProperty( "hazelcast.diagnostics.metric.level", "info" );
        System.setProperty( "hazelcast.diagnostics.invocation.sample.period.seconds", "30" );
        System.setProperty( "hazelcast.diagnostics.pending.invocations.period.seconds", "30" );
        System.setProperty( "hazelcast.diagnostics.slowoperations.period.seconds", "30" );
        System.setProperty( "hazelcast.diagnostics.storeLatency.period.seconds", "60" );
        System.setProperty( "hazelcast.diagnostics.directory", "D:/Parth/Hazelcast/logs/" );
        System.setProperty( "hazelcast.diagnostics.filename.prefix", "Server" );
        
        
        config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
        
        config.getNetworkConfig().getInterfaces().addInterface("127.0.0.1");
        
        JoinConfig join = config.getNetworkConfig().getJoin();
        join.getTcpIpConfig().addMember("192.168.0.104");
        join.getTcpIpConfig().setEnabled(true);

When trying to connect client it gives:

2021-07-22 12:07:24.081  INFO 192584 --- [ration.thread-0] c.h.c.i.p.t.AuthenticationMessageTask    : [192.168.0.104]:5701 [APP] [4.2] Received auth from Connection[id=2, /127.0.0.1:5701->/127.0.0.1:5703, qualifier=null, endpoint=[127.0.0.1]:5703, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: 6bebb36c-b865-4fd5-aa12-d0f3ec82ae8d, client version: 4.2

When Connect 1 ==============================================================
6bebb36c-b865-4fd5-aa12-d0f3ec82ae8d
8c54b788-7dfb-401a-ae2c-70444534fa8d
When Connect 1 ==============================================================
2021-07-22 12:07:38.902  WARN 192584 --- [.IO.thread-in-1] c.h.i.server.tcp.TcpServerConnection     : [192.168.0.104]:5701 [APP] [4.2] Connection[id=1, /127.0.0.1:5701->/127.0.0.1:5702, qualifier=null, endpoint=[127.0.0.1]:5702, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Exception in Connection[id=1, /127.0.0.1:5701->/127.0.0.1:5702, qualifier=null, endpoint=[127.0.0.1]:5702, alive=true, connectionType=JVM, planeIndex=-1], thread=hz.APP1.IO.thread-in-1

java.io.IOException: An existing connection was forcibly closed by the remote host
    at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[na:1.8.0_275]
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) ~[na:1.8.0_275]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_275]
    at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_275]
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[na:1.8.0_275]
    at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:119) ~[hazelcast-all-4.2.jar:4.2]
    at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383) [hazelcast-all-4.2.jar:4.2]
    at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368) [hazelcast-all-4.2.jar:4.2]
    at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294) [hazelcast-all-4.2.jar:4.2]
    at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249) [hazelcast-all-4.2.jar:4.2]

Client Configuration:

ClientConfig clientConfig = new ClientConfig();
        clientConfig.setClusterName("APP");
        clientConfig.getNetworkConfig().addAddress("localhost");
        clientConfig.getNetworkConfig().setSmartRouting(true);
        clientConfig.getNetworkConfig().addOutboundPortDefinition(5702-5720);
        
        
        clientConfig.setProperty("hazelcast.client.heartbeat.timeout", "60000");
        clientConfig.setProperty("hazelcast.client.heartbeat.timeout", "5000");
        
        ClientConnectionStrategyConfig connectionStrategyConfig = clientConfig.getConnectionStrategyConfig();
        connectionStrategyConfig.setReconnectMode(ClientConnectionStrategyConfig.ReconnectMode.ON);
        connectionRetryConfig.setInitialBackoffMillis(1000)
                             .setMaxBackoffMillis(60000)
                             .setMultiplier(2)
                             .setClusterConnectTimeoutMillis(1000)
                             .setJitter(0.2);

Let me know what wrong am I doing or what is best configuration for Member and client clustering where Member will be 1 and clients will be multiple.

Upvotes: 0

Views: 2330

Answers (1)

Vitaly Roslov
Vitaly Roslov

Reputation: 323

It's likely that the heartbeat timeout for the client is too short.

You've probably copied the heartbeat configuration from https://docs.hazelcast.com/imdg/4.2/clients/java.html

clientConfig.setProperty("hazelcast.client.heartbeat.timeout", "60000");
clientConfig.setProperty("hazelcast.client.heartbeat.interval", "5000");

but for the second line you've set timeout again instead of interval, overriding the value of 60000 with 5000

Upvotes: 1

Related Questions