Session Replication in Wildfly 10.1

Question

I am trying to enable session replication in my Wildfly 10.1 application with distributable WARs.

I am running on 2 instances of RedHat 7.2 on a managed host provider with full access to the OS and firewall. I don't have access to the router which our traffic is served, however the host has confirmed that multicast UDP is enabled.

I have SeLinux set to minimum, the ports are open in iptables, the multicast IPs have been subscribed, and my wildfly domain mode configuration is using a cloned full-ha profile with full-ha-sockets:

Here is the domain profile, which is vanilla with the exception of datasources:

<profile name="ha-dev2">
    <subsystem xmlns="urn:jboss:domain:logging:3.0">
        <add-logging-api-dependencies value="false"/>
        <console-handler name="CONSOLE">
            <level name="INFO"/>
            <formatter>
                <named-formatter name="COLOR-PATTERN"/>
            </formatter>
        </console-handler>
        <periodic-rotating-file-handler name="FILE" autoflush="true">
            <formatter>
                <named-formatter name="PATTERN"/>
            </formatter>
            <file relative-to="jboss.server.log.dir" path="server.log"/>
            <suffix value=".yyyy-MM-dd"/>
            <append value="true"/>
        </periodic-rotating-file-handler>
        <logger category="com.arjuna">
            <level name="WARN"/>
        </logger>
        <logger category="org.jboss.as.config">
            <level name="DEBUG"/>
        </logger>
        <logger category="sun.rmi">
            <level name="WARN"/>
        </logger>
        <root-logger>
            <level name="INFO"/>
            <handlers>
                <handler name="CONSOLE"/>
                <handler name="FILE"/>
            </handlers>
        </root-logger>
        <formatter name="PATTERN">
            <pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
        </formatter>
        <formatter name="COLOR-PATTERN">
            <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
        </formatter>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:batch-jberet:1.0">
        <default-job-repository name="in-memory"/>
        <default-thread-pool name="batch"/>
        <job-repository name="in-memory">
            <in-memory/>
        </job-repository>
        <thread-pool name="batch">
            <max-threads count="10"/>
            <keepalive-time time="30" unit="seconds"/>
        </thread-pool>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:bean-validation:1.0"/>
    <subsystem xmlns="urn:jboss:domain:datasources:4.0">
        <datasources>
            <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
                <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
                <driver>h2</driver>
                <security>
                    <user-name>sa</user-name>
                    <password>sa</password>
                </security>
            </datasource>
            <!-- DATASOURCES REDACTED -->
            <drivers>
                <driver name="h2" module="com.h2database.h2">
                    <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
                </driver>
            </drivers>
        </datasources>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:ee:4.0">
        <spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>
        <concurrent>
            <context-services>
                <context-service name="default" jndi-name="java:jboss/ee/concurrency/context/default" use-transaction-setup-provider="true"/>
            </context-services>
            <managed-thread-factories>
                <managed-thread-factory name="default" jndi-name="java:jboss/ee/concurrency/factory/default" context-service="default"/>
            </managed-thread-factories>
            <managed-executor-services>
                <managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" keepalive-time="5000"/>
            </managed-executor-services>
            <managed-scheduled-executor-services>
                <managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" keepalive-time="3000"/>
            </managed-scheduled-executor-services>
        </concurrent>
        <default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/ExampleDS" jms-connection-factory="java:jboss/DefaultJMSConnectionFactory" managed-executor-service="java:jboss/ee/concurrency/executor/default" managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:ejb3:4.0">
        <session-bean>
            <stateless>
                <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
            </stateless>
            <stateful default-access-timeout="5000" cache-ref="distributable" passivation-disabled-cache-ref="simple"/>
            <singleton default-access-timeout="5000"/>
        </session-bean>
        <mdb>
            <resource-adapter-ref resource-adapter-name="${ejb.resource-adapter-name:activemq-ra.rar}"/>
            <bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
        </mdb>
        <pools>
            <bean-instance-pools>
                <strict-max-pool name="slsb-strict-max-pool" derive-size="from-worker-pools" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                <strict-max-pool name="mdb-strict-max-pool" derive-size="from-cpu-count" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
            </bean-instance-pools>
        </pools>
        <caches>
            <cache name="simple"/>
            <cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>
        </caches>
        <passivation-stores>
            <passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>
        </passivation-stores>
        <async thread-pool-name="default"/>
        <timer-service thread-pool-name="default" default-data-store="default-file-store">
            <data-stores>
                <file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>
            </data-stores>
        </timer-service>
        <remote connector-ref="http-remoting-connector" thread-pool-name="default"/>
        <thread-pools>
            <thread-pool name="default">
                <max-threads count="10"/>
                <keepalive-time time="100" unit="milliseconds"/>
            </thread-pool>
        </thread-pools>
        <iiop enable-by-default="false" use-qualified-name="false"/>
        <default-security-domain value="other"/>
        <default-missing-method-permissions-deny-access value="true"/>
        <log-system-exceptions value="true"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:io:1.1">
        <worker name="default"/>
        <buffer-pool name="default"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:infinispan:4.0">
        <cache-container name="server" aliases="singleton cluster" default-cache="default" module="org.wildfly.clustering.server">
            <transport lock-timeout="60000"/>
            <replicated-cache name="default" mode="SYNC">
                <transaction mode="BATCH"/>
            </replicated-cache>
        </cache-container>
        <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
            <transport lock-timeout="60000"/>
            <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
                <locking isolation="REPEATABLE_READ"/>
                <transaction mode="BATCH"/>
                <file-store/>
            </distributed-cache>
            <distributed-cache name="concurrent" mode="SYNC" l1-lifespan="0" owners="2">
                <file-store/>
            </distributed-cache>
        </cache-container>
        <cache-container name="ejb" aliases="sfsb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan">
            <transport lock-timeout="60000"/>
            <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
                <locking isolation="REPEATABLE_READ"/>
                <transaction mode="BATCH"/>
                <file-store/>
            </distributed-cache>
        </cache-container>
        <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">
            <transport lock-timeout="60000"/>
            <local-cache name="local-query">
                <eviction strategy="LRU" max-entries="10000"/>
                <expiration max-idle="100000"/>
            </local-cache>
            <invalidation-cache name="entity" mode="SYNC">
                <transaction mode="NON_XA"/>
                <eviction strategy="LRU" max-entries="10000"/>
                <expiration max-idle="100000"/>
            </invalidation-cache>
            <replicated-cache name="timestamps" mode="ASYNC"/>
        </cache-container>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:iiop-openjdk:1.0">
        <orb socket-binding="iiop" ssl-socket-binding="iiop-ssl"/>
        <initializers security="identity" transactions="spec"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
    <subsystem xmlns="urn:jboss:domain:jca:4.0">
        <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
        <bean-validation enabled="true"/>
        <default-workmanager>
            <short-running-threads>
                <core-threads count="50"/>
                <queue-length count="50"/>
                <max-threads count="50"/>
                <keepalive-time time="10" unit="seconds"/>
            </short-running-threads>
            <long-running-threads>
                <core-threads count="50"/>
                <queue-length count="50"/>
                <max-threads count="50"/>
                <keepalive-time time="10" unit="seconds"/>
            </long-running-threads>
        </default-workmanager>
        <cached-connection-manager/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
    <subsystem xmlns="urn:jboss:domain:jgroups:4.0">
        <channels default="ee">
            <channel name="ee" stack="udp"/>
        </channels>
        <stacks>
            <stack name="udp">
                <transport type="UDP" socket-binding="jgroups-udp"/>
                <protocol type="PING"/>
                <protocol type="MERGE3"/>
                <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
                <protocol type="FD_ALL"/>
                <protocol type="VERIFY_SUSPECT"/>
                <protocol type="pbcast.NAKACK2"/>
                <protocol type="UNICAST3"/>
                <protocol type="pbcast.STABLE"/>
                <protocol type="pbcast.GMS"/>
                <protocol type="UFC"/>
                <protocol type="MFC"/>
                <protocol type="FRAG2"/>
            </stack>
            <stack name="tcp">
                <transport type="TCP" socket-binding="jgroups-tcp"/>
                <protocol type="MPING" socket-binding="jgroups-mping"/>
                <protocol type="MERGE3"/>
                <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                <protocol type="FD"/>
                <protocol type="VERIFY_SUSPECT"/>
                <protocol type="pbcast.NAKACK2"/>
                <protocol type="UNICAST3"/>
                <protocol type="pbcast.STABLE"/>
                <protocol type="pbcast.GMS"/>
                <protocol type="MFC"/>
                <protocol type="FRAG2"/>
            </stack>
        </stacks>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:jmx:1.3">
        <expose-resolved-model/>
        <expose-expression-model/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:jpa:1.1">
        <jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:jsf:1.0"/>
    <subsystem xmlns="urn:jboss:domain:jsr77:1.0"/>
    <subsystem xmlns="urn:jboss:domain:mail:2.0">
        <mail-session name="default" jndi-name="java:jboss/mail/Default">
            <smtp-server outbound-socket-binding-ref="mail-smtp"/>
        </mail-session>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
        <server name="default">
            <cluster password="${jboss.messaging.cluster.password:@password@}"/>
            <bindings-directory/>
            <journal-directory/>
            <large-messages-directory/>
            <paging-directory/>
            <security-setting name="#">
                <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
            </security-setting>
            <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>
            <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
            <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
                <param name="batch-delay" value="50"/>
            </http-connector>
            <in-vm-connector name="in-vm" server-id="0"/>
            <http-acceptor name="http-acceptor" http-listener="default"/>
            <http-acceptor name="http-acceptor-throughput" http-listener="default">
                <param name="batch-delay" value="50"/>
                <param name="direct-deliver" value="false"/>
            </http-acceptor>
            <in-vm-acceptor name="in-vm" server-id="0"/>
            <broadcast-group name="bg-group1" jgroups-channel="activemq-cluster" connectors="http-connector"/>
            <discovery-group name="dg-group1" jgroups-channel="activemq-cluster"/>
            <cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1"/>
            <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
            <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
            <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
            <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" block-on-acknowledge="true" reconnect-attempts="-1"/>
            <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa"/>
        </server>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:modcluster:2.0">
        <mod-cluster-config advertise-socket="modcluster" balancer="dev-ha-server-group" connector="ajp">
            <dynamic-load-provider>
                <load-metric type="busyness"/>
            </dynamic-load-provider>
        </mod-cluster-config>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:naming:2.0">
        <remote-naming/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
    <subsystem xmlns="urn:jboss:domain:remoting:3.0">
        <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:resource-adapters:4.0"/>
    <subsystem xmlns="urn:jboss:domain:request-controller:1.0"/>
    <subsystem xmlns="urn:jboss:domain:sar:1.0"/>
    <subsystem xmlns="urn:jboss:domain:security:1.2">
        <security-domains>
            <security-domain name="other" cache-type="default">
                <authentication>
                    <login-module code="Remoting" flag="optional">
                        <module-option name="password-stacking" value="useFirstPass"/>
                    </login-module>
                    <login-module code="RealmDirect" flag="required">
                        <module-option name="password-stacking" value="useFirstPass"/>
                    </login-module>
                </authentication>
            </security-domain>
            <security-domain name="jboss-web-policy" cache-type="default">
                <authorization>
                    <policy-module code="Delegating" flag="required"/>
                </authorization>
            </security-domain>
            <security-domain name="jboss-ejb-policy" cache-type="default">
                <authorization>
                    <policy-module code="Delegating" flag="required"/>
                </authorization>
            </security-domain>
            <security-domain name="jaspitest" cache-type="default">
                <authentication-jaspi>
                    <login-module-stack name="dummy">
                        <login-module code="Dummy" flag="optional"/>
                    </login-module-stack>
                    <auth-module code="Dummy"/>
                </authentication-jaspi>
            </security-domain>
        </security-domains>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:security-manager:1.0">
        <deployment-permissions>

            <maximum-set>
                <permission class="java.security.AllPermission"/>
            </maximum-set>
        </deployment-permissions>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:singleton:1.0">
        <singleton-policies default="default">
            <singleton-policy name="default" cache-container="server">
                <simple-election-policy/>
            </singleton-policy>
        </singleton-policies>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:transactions:3.0">
        <core-environment>
            <process-id>
                <uuid/>
            </process-id>
        </core-environment>
        <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:undertow:3.1">
        <buffer-cache name="default"/>
        <server name="default-server">
            <ajp-listener name="ajp" socket-binding="ajp" max-post-size="26214400"/>
            <http-listener name="default" socket-binding="http" max-post-size="26214400" redirect-socket="https" enable-http2="true"/>
            <https-listener name="https" socket-binding="https" max-post-size="26214400" security-realm="ApplicationRealm" enable-http2="true"/>
            <host name="default-host" alias="localhost">
                <location name="/" handler="welcome-content"/>
                <filter-ref name="server-header"/>
                <filter-ref name="x-powered-by-header"/>
                <filter-ref name="gzipFilter" predicate="exists['%{o,Content-Type}'] and regex[pattern='(?:application/javascript|text/css|text/html|text/xml|application/json)(;.*)?', value=%{o,Content-Type}, full-match=true]"/>
            </host>
        </server>
        <servlet-container name="default">
            <jsp-config trim-spaces="true"/>
            <websockets/>
        </servlet-container>
        <handlers>
            <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
        </handlers>
        <filters>
            <response-header name="server-header" header-name="Server" header-value="WildFly/10"/>
            <response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>
            <gzip name="gzipFilter"/>
        </filters>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:webservices:2.0">
        <wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>
        <endpoint-config name="Standard-Endpoint-Config"/>
        <endpoint-config name="Recording-Endpoint-Config">
            <pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">
                <handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>
            </pre-handler-chain>
        </endpoint-config>
        <client-config name="Standard-Client-Config"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:weld:3.0"/>
</profile>

This the socket binding group:

    <socket-binding-group name="dev-full-ha-sockets" default-interface="public">
        <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
        <socket-binding name="http" port="${jboss.http.port:8080}"/>
        <socket-binding name="https" port="${jboss.https.port:18443}"/>
        <socket-binding name="iiop" interface="unsecure" port="3528"/>
        <socket-binding name="iiop-ssl" interface="unsecure" port="3529"/>
        <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
        <socket-binding name="jgroups-tcp" port="7600"/>
        <socket-binding name="jgroups-tcp-fd" port="57600"/>
        <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" port="54200"/>
        <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <outbound-socket-binding name="mail-smtp">
            <remote-destination host="localhost" port="25"/>
        </outbound-socket-binding>
    </socket-binding-group>

And finally, the server-group:

<server-group name="dev-ha-server-group" profile="ha-dev2">
            <jvm name="default">
                <heap size="256m" max-size="1024m"/>
            </jvm>
            <socket-binding-group ref="dev-full-ha-sockets"/>
            <deployments>
             <!-- Deployments redacted -->
            </deployments>
        </server-group>

Using TCPDump, I can see traffic for modcluster from both of my servers.

I have also run the McastReceiverTest (src: https://github.com/belaban/JGroups/blob/master/tests/other/org/jgroups/tests/McastReceiverTest.java) via:

java org.jgroups.tests.McastReceiverTest -mcast_addr 230.0.0.4 -port 45700

and have used the following command:

printf "GET / HTTP/1.0\r\n\r\n" | nc -vu 230.0.0.4 45700

and see the UDP traffic coming through.

I am not seeing anything for the jgroups from wildfly though. In my logs, when I look for the subscribed channels, I see

"Received new cluster view for channel server: [app-one:server-one|0] (1) [app-one:server-one]"

But I never see my second server on either instance. When I test, I have an F5 load balancer in front of both RedHat instances.

I have a few assumptions that I am hoping to clear up:

1. Do I need to have Apache enabled for Modcluster? I'd assume not as it appears to be used for load balancing if you don't have a separate balancing mechanism.

2. Is Session Replication the same as running a "hot-hot" scenario so that 2 applications can run and keep a session alive? My understanding is that I should be able to log into the application on server A and server B would be able to keep that session alive as well. We are trying to prevent the need for session stickyness.

3. Can this run in a multi-server environment or is it only for 2 instances on the same local machine? I would assume that it should not matter if this is across a local machine or on 2 separate physicals/vms behind the same firewall.

Thank you.

---

I found a solution which has been eluding me. The problem with my configuration was in my bind IP. I was using 127.0.0.1 for my jboss.bind.address as I had Apache in front of my Wildfly configuration. I changed the bind IP to the IP of the NIC (in my case it was a 10.* address).

Answer 1

The problem with my configuration was in my bind IP. I was using 127.0.0.1 for my jboss.bind.address as I had Apache in front of my Wildfly configuration. I changed the bind IP to the IP of the NIC (in my case it was a 10.* address).

Answer 2

It would be helpful if you posted your entire standalone.xml file.

To Answer your questions:

1. No, you can use any load balancer. Mod cluster is just nice because Wildfly talks directly to it so servers are removed quicker when shutting down.

2. yes. With replication all session data is stored in Infinispan which is then replicated to other servers in the cluster.

3. The primary use case for clustering is to have instances on different servers.

Because I don't have your entire configuration and don't know anything about your hosting here is my best guesses for what is wrong in order I would check:

• You are running in AWS - UDP multicast doesn't work in AWS. you must use another ping method.
• Your server is set up to not accept UDP multicast broadcast. Check all firewall and networking rules.
• Your configuration has a flaw in it. If you post entire configuration I can take a look.
• your web.xml file is missing a <distributable/> tag