Jboss-CLI won't connect from remote

Question

I have been trying to set up remote deployment of a ear using maven, but it fails because the jboss-cli.bat script cannot connect to the server. It fails with the stack trace:

C:\Users\pstephens\Programming\jboss-eap-6.2\jboss-eap-6.2\bin>jboss-cli.bat --controller=172.22.2.29:9999 --connect --user=pstephens --password=#######
org.jboss.as.cli.CliInitializationException: Failed to connect to the controller
    at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:284)
    at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:262)
    at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.jboss.modules.Module.run(Module.java:292)
    at org.jboss.modules.Main.main(Main.java:455)
Caused by: org.jboss.as.cli.CommandLineException: The controller is not available at 172.22.2.29:9990
    at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:969)
    at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:808)
    at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:784)
    at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:282)
    ... 8 more
Caused by: java.io.IOException: java.net.ConnectException: JBAS012144: Could not connect to remote://172.22.2.29:9990. The connection timed out
    at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:129)
    at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:71)
    at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:947)
    ... 11 more
Caused by: java.net.ConnectException: JBAS012144: Could not connect to remote://172.22.2.29:9990. The connection timed out
    at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:131)
    at org.jboss.as.protocol.ProtocolConnectionManager$EstablishingConnection.connect(ProtocolConnectionManager.java:256)
    at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:70)
    at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:204)
    at org.jboss.as.cli.impl.CLIModelControllerClient.getOrCreateChannel(CLIModelControllerClient.java:160)
    at org.jboss.as.cli.impl.CLIModelControllerClient$2.getChannel(CLIModelControllerClient.java:120)
    at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:117)
    at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:92)
    at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:236)
    at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:141)
    at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127)
    ... 13 more

My standalone.xml has the following settings:

<interfaces>
    <interface name="management">
        <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
    </interface>
    <interface name="public">
        <inet-address value="${jboss.bind.address:127.0.0.1}"/>
    </interface>
    <interface name="unsecure">
        <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
    </interface>
</interfaces>

<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
    <socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>
    <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
    <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9443}"/>
    <socket-binding name="ajp" port="8009"/>
    <socket-binding name="http" port="8080"/>
    <socket-binding name="https" port="8443"/>
    <socket-binding name="remoting" port="4447"/>
    <socket-binding name="txn-recovery-environment" port="4712"/>
    <socket-binding name="txn-status-manager" port="4713"/>
    <outbound-socket-binding name="mail-smtp">
        <remote-destination host="localhost" port="25"/>
    </outbound-socket-binding>
</socket-binding-group>

which are fairly standard and I start up with the command:

(sudo) ./standalone.sh -b=172.22.2.29 -bmanagement=172.22.2.29

which binds the jboss instance to the server ip. The cli works fine if run locally on the server, but fails when run remotely. I have extensively investigated firewall and port issues and can see nothing untoward. nmap confirms that 172.22.2.29:9999 is open and visible from my machine. When the connection is attempted there is no log entry from the jboss server at all.

Things I have tried: (1) Examining Firewall logs to look for blocked connection, I am satisfied that there is no problem with the firewalls or port settings. (2) Using clean installs of various bits and pieces including JRE's, jboss etc. (3) Changing the port bindings in case there was a conflict with some as yet unidentified process.

The only other interesting clue I have discovered is that if I watch the firewall logs on the server in real time, they show a TCP connection received and established, only to immediately receive a tear-down request, whereas watching netstat on my machine shows that it believes that the connection was established with port 9999. This continues as long as jboss-cli is running, even after it has thrown an exception.

I am hoping there is some explanation that I have overlooked, and one of you helpful people can point it out to me.

Rest of standalone.xml below:

<management>
    <security-realms>
        <security-realm name="ManagementRealm">
            <authentication>
                <local default-user="$local"/>
                <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
            </authentication>
            <authorization map-groups-to-roles="false">
                <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
            </authorization>
        </security-realm>
        <security-realm name="ApplicationRealm">
            <authentication>
                <local default-user="$local" allowed-users="*"/>
                <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
            </authentication>
            <authorization>
                <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
            </authorization>
        </security-realm>
    </security-realms>
    <audit-log>
        <formatters>
            <json-formatter name="json-formatter"/>
        </formatters>
        <handlers>
            <file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>
        </handlers>
        <logger log-boot="true" log-read-only="false" enabled="false">
            <handlers>
                <handler name="file"/>
            </handlers>
        </logger>
    </audit-log>
    <management-interfaces>
        <native-interface security-realm="ManagementRealm">
            <socket-binding native="management-native"/>
        </native-interface>
        <http-interface security-realm="ManagementRealm">
            <socket-binding http="management-http"/>
        </http-interface>
    </management-interfaces>
    <access-control provider="simple">
        <role-mapping>
            <role name="SuperUser">
                <include>
                    <user name="$local"/>
                    <user realm="ManagementRealm" name="pstephens"/>
                </include>
            </role>
            <role name="Administrator">
                <include>
                    <user realm="ManagementRealm" name="pstephens"/>
                </include>
            </role>
            <role name="Auditor">
                <include>
                    <user realm="ManagementRealm" name="pstephens"/>
                </include>
            </role>
            <role name="Deployer">
                <include>
                    <user realm="ManagementRealm" name="pstephens"/>
                </include>
            </role>
            <role name="Operator">
                <include>
                    <user realm="ManagementRealm" name="pstephens"/>
                </include>
            </role>
            <role name="Monitor">
                <include>
                    <user realm="ManagementRealm" name="pstephens"/>
                </include>
            </role>
            <role name="Maintainer">
                <include>
                    <user realm="ManagementRealm" name="pstephens"/>
                </include>
            </role>
        </role-mapping>
    </access-control>
</management>

<profile>
    <subsystem xmlns="urn:jboss:domain:logging:1.3">
        <console-handler name="CONSOLE">
            <level name="DEBUG"/>
            <formatter>
                <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>
            </formatter>
        </console-handler>
        <custom-handler name="com.bmt.logging" class="com.bmt.logging.JdbcHandler" module="com.bmt.logging">
            <level name="DEBUG"/>
            <formatter>
                <pattern-formatter pattern="%d{yyyy/MM/dd-HH:mm:ss.SSS} $%p $%c $%t $%X{ip} $%s"/>
            </formatter>
            <properties>
                <property name="driverClassName" value="org.postgresql.Driver"/>
                <property name="insertStatement" value="INSERT INTO logtable (timestamp, loglevel, logcat, threadname, callingip, message) VALUES (?, ?, ?, ?, ?, ?)"/>
                <property name="password" value="########"/>
                <property name="username" value="postgres"/>
                <property name="jdbcUrl" value="jdbc:postgresql://172.22.2.29:5432/ContainSssLog"/>
            </properties>
        </custom-handler>
        <periodic-rotating-file-handler name="FILE" autoflush="true">
            <formatter>
                <pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>
            </formatter>
            <file relative-to="jboss.server.log.dir" path="server.log"/>
            <suffix value=".yyyy-MM-dd"/>
            <append value="true"/>
        </periodic-rotating-file-handler>
        <logger category="com.arjuna">
            <level name="WARN"/>
        </logger>
        <logger category="org.apache.tomcat.util.modeler">
            <level name="WARN"/>
        </logger>
        <logger category="org.jboss.as.config">
            <level name="DEBUG"/>
        </logger>
        <logger category="sun.rmi">
            <level name="WARN"/>
        </logger>
        <logger category="jacorb">
            <level name="WARN"/>
        </logger>
        <logger category="jacorb.config">
            <level name="ERROR"/>
        </logger>
        <root-logger>
            <level name="INFO"/>
            <handlers>
                <handler name="CONSOLE"/>
                <handler name="FILE"/>
                <handler name="com.bmt.logging"/>
            </handlers>
        </root-logger>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:datasources:1.1">
        <datasources>
            <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
                <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
                <driver>h2</driver>
                <security>
                    <user-name>sa</user-name>
                    <password>sa</password>
                </security>
            </datasource>
            <drivers>
                <driver name="h2" module="com.h2database.h2">
                    <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
                </driver>
            </drivers>
        </datasources>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">
        <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:ee:1.1">
        <spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>
        <jboss-descriptor-property-replacement>true</jboss-descriptor-property-replacement>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:ejb3:1.4">
        <session-bean>
            <stateless>
                <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
            </stateless>
            <stateful default-access-timeout="5000" cache-ref="simple"/>
            <singleton default-access-timeout="5000"/>
        </session-bean>
        <pools>
            <bean-instance-pools>
                <strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                <strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
            </bean-instance-pools>
        </pools>
        <caches>
            <cache name="simple" aliases="NoPassivationCache"/>
            <cache name="passivating" passivation-store-ref="file" aliases="SimpleStatefulCache"/>
        </caches>
        <passivation-stores>
            <file-passivation-store name="file"/>
        </passivation-stores>
        <async thread-pool-name="default"/>
        <timer-service thread-pool-name="default">
            <data-store path="timer-service-data" relative-to="jboss.server.data.dir"/>
        </timer-service>
        <remote connector-ref="remoting-connector" thread-pool-name="default"/>
        <thread-pools>
            <thread-pool name="default">
                <max-threads count="10"/>
                <keepalive-time time="100" unit="milliseconds"/>
            </thread-pool>
        </thread-pools>
        <default-security-domain value="other"/>
        <default-missing-method-permissions-deny-access value="true"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:infinispan:1.4">
        <cache-container name="web" aliases="standard-session-cache" default-cache="local-web" module="org.jboss.as.clustering.web.infinispan">
            <local-cache name="local-web" batching="true">
                <file-store passivation="false" purge="false"/>
            </local-cache>
        </cache-container>
        <cache-container name="hibernate" default-cache="local-query" module="org.jboss.as.jpa.hibernate:4">
            <local-cache name="entity">
                <transaction mode="NON_XA"/>
                <eviction strategy="LRU" max-entries="10000"/>
                <expiration max-idle="100000"/>
            </local-cache>
            <local-cache name="local-query">
                <transaction mode="NONE"/>
                <eviction strategy="LRU" max-entries="10000"/>
                <expiration max-idle="100000"/>
            </local-cache>
            <local-cache name="timestamps">
                <transaction mode="NONE"/>
                <eviction strategy="NONE"/>
            </local-cache>
        </cache-container>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
    <subsystem xmlns="urn:jboss:domain:jca:1.1">
        <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
        <bean-validation enabled="true"/>
        <default-workmanager>
            <short-running-threads>
                <core-threads count="50"/>
                <queue-length count="50"/>
                <max-threads count="50"/>
                <keepalive-time time="10" unit="seconds"/>
            </short-running-threads>
            <long-running-threads>
                <core-threads count="50"/>
                <queue-length count="50"/>
                <max-threads count="50"/>
                <keepalive-time time="10" unit="seconds"/>
            </long-running-threads>
        </default-workmanager>
        <cached-connection-manager/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
    <subsystem xmlns="urn:jboss:domain:jmx:1.3">
        <expose-resolved-model/>
        <expose-expression-model/>
        <remoting-connector/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:jpa:1.1">
        <jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:jsf:1.0"/>
    <subsystem xmlns="urn:jboss:domain:mail:1.1">
        <mail-session jndi-name="java:jboss/mail/Default">
            <smtp-server outbound-socket-binding-ref="mail-smtp"/>
        </mail-session>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:naming:1.4">
        <remote-naming/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
    <subsystem xmlns="urn:jboss:domain:remoting:1.1">
        <connector name="remoting-connector" socket-binding="remoting" security-realm="ApplicationRealm"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:resource-adapters:1.1"/>
    <subsystem xmlns="urn:jboss:domain:sar:1.0"/>
    <subsystem xmlns="urn:jboss:domain:security:1.2">
        <security-domains>
            <security-domain name="other" cache-type="default">
                <authentication>
                    <login-module code="Remoting" flag="optional">
                        <module-option name="password-stacking" value="useFirstPass"/>
                    </login-module>
                    <login-module code="RealmDirect" flag="required">
                        <module-option name="password-stacking" value="useFirstPass"/>
                    </login-module>
                </authentication>
            </security-domain>
            <security-domain name="jboss-web-policy" cache-type="default">
                <authorization>
                    <policy-module code="Delegating" flag="required"/>
                </authorization>
            </security-domain>
            <security-domain name="jboss-ejb-policy" cache-type="default">
                <authorization>
                    <policy-module code="Delegating" flag="required"/>
                </authorization>
            </security-domain>
        </security-domains>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:threads:1.1"/>
    <subsystem xmlns="urn:jboss:domain:transactions:1.4">
        <core-environment>
            <process-id>
                <uuid/>
            </process-id>
        </core-environment>
        <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
        <coordinator-environment default-timeout="300"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:web:1.5" default-virtual-server="default-host" native="false">
        <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
        <virtual-server name="default-host" enable-welcome-root="true">
            <alias name="localhost"/>
            <alias name="example.com"/>
        </virtual-server>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:webservices:1.2">
        <modify-wsdl-address>true</modify-wsdl-address>
        <wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>
        <endpoint-config name="Standard-Endpoint-Config"/>
        <endpoint-config name="Recording-Endpoint-Config">
            <pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">
                <handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>
            </pre-handler-chain>
        </endpoint-config>
        <client-config name="Standard-Client-Config"/>
    </subsystem>
    <subsystem xmlns="urn:jboss:domain:weld:1.0"/>
</profile>

Answer 1

The cause of this error was that the system clock on the ubuntu server was wrong. This caused the tcp protocol itself to tear down the connection, for reasons that remain obscure, but probably because the server judged that the connection had been delayed en route and that the client was no longer listening on the other end.

Using the --timeout= flag can help with this error also, as the default timeout of 5 seconds is very short for external handshakes.