Cannot connect to RabbitMQ in Cloudfoundry Micro Instance

Question

I have an application I have been running on cloudfoundry.com for a while and everything works great when deployed to api.cloudfoundry.com. I recently set up a CF Micro instance to do local testing in a cloud environment and when my application is deployed to the micro instance, it cannot connect to my rabbit service. I get the following exception:

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.integration.MessageHandlingException: error occurred in message handler [org.springframework.integration.amqp.outbound.AmqpOutboundEndpoint#0]
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:894)
    org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:789)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
    org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
    org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
...
org.springframework.integration.MessageHandlingException: error occurred in message handler [org.springframework.integration.amqp.outbound.AmqpOutboundEndpoint#0]
    org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:79)
    org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115)
    org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102)
...
org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection timed out
    org.springframework.amqp.rabbit.connection.RabbitUtils.convertRabbitAccessException(RabbitUtils.java:109)
    org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:163)
    org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:228)
    org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:119)
    org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:163)
    org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:109)
    org.springframework.amqp.rabbit.connection.RabbitAccessor.getTransactionalResourceHolder(RabbitAccessor.java:100)
    org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:591)
    org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:329)
    org.springframework.amqp.rabbit.core.RabbitTemplate.convertAndSend(RabbitTemplate.java:385)
    org.springframework.integration.amqp.outbound.AmqpOutboundEndpoint.send(AmqpOutboundEndpoint.java:200)
    org.springframework.integration.amqp.outbound.AmqpOutboundEndpoint.handleRequestMessage(AmqpOutboundEndpoint.java:192)

I'm not sure what to look at (i'm sure there is something in the micro instance I can look at, but not sure what), so I thought I'd ask here.

I have verified that the service exists:

vmc services
Getting services... OK

name                    service    version
xxx-mongo    mongodb    2.0    
xxx-mysql    mysql      5.1    
xxx-rabbit   rabbitmq   2.4    

I have verified that the service is bound to my application:

vmc app myapp
myapp: running
  platform: spring on java
  usage: 512M × 1 instance
  urls: mydomain.cloudfoundry.me
  services: xxx-mysql, xxx-rabbit, xxx-mongo

I even tried to rebind it but it says it is bound:

vmc bind-service xxx-rabbit myapp
Binding xxx-rabbit to myapp... SKIPPED
App myapp already binds xxx-rabbit.

My app is a Spring application. The config for the rabbit service is set up like so:

<?xml version="1.0" encoding="UTF-8"?>
<beans  xmlns="http://www.springframework.org/schema/beans"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:cloud="http://schema.cloudfoundry.org/spring"
        xmlns:jdbc="http://www.springframework.org/schema/jdbc"
        xmlns:util="http://www.springframework.org/schema/util"
        xmlns:mongo="http://www.springframework.org/schema/data/mongo"
        xmlns:rabbit="http://www.springframework.org/schema/rabbit"
        xsi:schemaLocation="http://www.springframework.org/schema/data/mongo
          http://www.springframework.org/schema/data/mongo/spring-mongo-1.1.xsd
          http://www.springframework.org/schema/rabbit http://www.springframework.org/schema/rabbit/spring-rabbit-1.0.xsd
          http://www.springframework.org/schema/jdbc
          http://www.springframework.org/schema/jdbc/spring-jdbc-3.1.xsd
          http://schema.cloudfoundry.org/spring
          http://schema.cloudfoundry.org/spring/cloudfoundry-spring.xsd
          http://www.springframework.org/schema/beans
          http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
          http://www.springframework.org/schema/util
          http://www.springframework.org/schema/util/spring-util-3.1.xsd">

    <!-- MongoDB setup -->
    <mongo:db-factory
        id="mongoDbFactory"
        dbname="xxx"
        host="127.0.0.1"
        port="27017"
        username="test_user"
        password="test_pass"  />

    <bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate">
        <constructor-arg ref="mongoDbFactory" />
    </bean>

    <!-- Set up the AmqpTemplate/RabbitTemplate: -->
    <rabbit:connection-factory id="rabbitConnectionFactory"/>
    <rabbit:template id="rabbitTemplate" connection-factory="rabbitConnectionFactory"/>

    <bean id="jdbcDataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
        <property name="driverClassName" value="com.mysql.jdbc.Driver" />
        <property name="url" value="jdbc:mysql://127.0.0.1:3306/test" />
        <property name="username" value="spring" />
        <property name="password" value="spring" />
    </bean>

    <jdbc:initialize-database data-source="jdbcDataSource">
        <jdbc:script location="classpath:oauthTokenStoreSchema.sql"/>
    </jdbc:initialize-database>
</beans>

This config relies on the Spring auto-reconfig, which is working fine as far as I can tell. It doesn't complain as startup, only when I actually try to send a message to the queue like so:

org.springframework.integration.Message<Message> msg = MessageBuilder.withPayload(m).build();
messagesChannel.send(msg);

Lastly, I logged into the micro instance to see if rabbit is actually running and it appears to be:

vcap@micro:~$ ps -ef | grep rabbit
root      1492     1  0 Jan14 ?        00:17:00 ruby /var/vcap/packages/rabbit_node/services/rabbit/bin/rabbit_node -c /var/vcap/jobs/rabbit_node/config/rabbit_node.yml
root      1509     1  0 Jan14 ?        00:14:48 ruby /var/vcap/packages/rabbit_gateway/services/ng/rabbit/bin/rabbit_gateway -c /var/vcap/jobs/rabbit_gateway/config/rabbit_gateway.yml
root      1771  1492  0 Jan14 ?        00:14:36 /var/vcap/data/packages/erlang/3/lib/erlang/erts-5.8.2/bin/beam -W w -K true -A30 -P 1048576 -- -root /var/vcap/data/packages/erlang/3/lib/erlang -progname erl -- -home /var/vcap/store/rabbit/instances/d55204b5-814c-4c64-bd35-861d77ee927d -- -noshell -noinput -sname d55204b5-814c-4c64-bd35-861d77ee927d@localhost -boot /var/vcap/store/rabbit/instances/d55204b5-814c-4c64-bd35-861d77ee927d/plugins/rabbit -config /var/vcap/store/rabbit/instances/d55204b5-814c-4c64-bd35-861d77ee927d/config/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -rabbit tcp_listeners [{"10.0.1.21",10001}] -sasl errlog_type error -kernel error_logger {file,"/var/vcap/sys/service-log/rabbit/d55204b5-814c-4c64-bd35-861d77ee927d/d55204b5-814c-4c64-bd35-861d77ee927d@localhost.log"} -sasl sasl_error_logger {file,"/var/vcap/sys/service-log/rabbit/d55204b5-814c-4c64-bd35-861d77ee927d/d55204b5-814c-4c64-bd35-861d77ee927d@localhost-sasl.log"} -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/vcap/store/rabbit/instances/d55204b5-814c-4c64-bd35-861d77ee927d/mnesia" -smp disable
vcap     49299 49278  0 02:38 pts/0    00:00:00 grep --color=auto rabbit

Again, when I deploy to api.cloudfoundry.com, the app can connect to the rabbitmq just fine. I'm running v119 of micro as well if that helps.

Any help you could provide would be great! Thanks.

EDIT: Per Hitesh recommendation, I tried to use the cloud profile, but that didn't work. I also tried this:

<beans profile="default">
        <rabbit:connection-factory
            id="rabbitConnectionFactory"
            host="localhost"
            password="testpwd"
            port="1238"
            username="testuser"
            virtual-host="virthost" />
        <rabbit:template id="rabbitTemplate" connection-factory="rabbitConnectionFactory"/>
    </beans>

    <beans profile="cloud">
        <cloud:rabbit-connection-factory id="rabbitConnectionFactory" />
        <rabbit:template id="rabbitTemplate" connection-factory="rabbitConnectionFactory"/>
    </beans>

What is interesting though, is that I tried this (copied out of the docs):

<rabbit:connection-factory
    id="rabbitConnectionFactory"
    host="localhost"
    password="testpwd"
    port="1238"
    username="testuser"
    virtual-host="virthost" />
<rabbit:template id="rabbitTemplate" connection-factory="rabbitConnectionFactory"/>

And now my sending application works! But my receiving app still doesn't work. It gets this error:

ERROR: main org.springframework.web.servlet.DispatcherServlet - Context initialization failed
org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter#0'; nested exception is org.springframework.amqp.UncategorizedAmqpException: java.util.concurrent.TimeoutException: Timed out waiting for startup
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:170)
    at org.springframework.context.support.DefaultLifecycleProcessor.access$1(DefaultLifecycleProcessor.java:154)
    at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:339)
    at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:143)
    at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:108)
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:926)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:467)
    at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:631)
    at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:588)
    at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:645)
    at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:508)
    at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:449)
    at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:133)
    at javax.servlet.GenericServlet.init(GenericServlet.java:212)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1206)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1026)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4421)
    at org.apache.catalina.core.StandardContext.start(StandardContext.java:4734)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
    at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1079)
    at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1002)
    at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:506)
    at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
    at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
    at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
    at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
    at org.apache.catalina.core.StandardService.start(StandardService.java:525)
    at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: org.springframework.amqp.UncategorizedAmqpException: java.util.concurrent.TimeoutException: Timed out waiting for startup
    at org.springframework.amqp.rabbit.connection.RabbitUtils.convertRabbitAccessException(RabbitUtils.java:118)
    at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:106)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:365)
    at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter.doStart(AmqpInboundChannelAdapter.java:85)
    at org.springframework.integration.endpoint.AbstractEndpoint.start(AbstractEndpoint.java:84)
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:167)
    ... 39 more
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for startup
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.getStartupException(SimpleMessageListenerContainer.java:512)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doStart(SimpleMessageListenerContainer.java:337)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:363)
    ... 42 more

I've tried all the configurations above, but none work.

Answer 1

I would suggest trying out the sample spring app using rabbitmq, it lives here : https://github.com/rabbitmq/rabbitmq-cloudfoundry-samples/tree/master/spring. Also there is a step by step doc to help you through it. You can compare the config there with your app and if it works, that means something might be missing in the config. Also if this does not help then you can post a question here "http://support.cloudfoundry.com" giving us access to your application so we can try deploying it ourselves.

Thanks,

• Hitesh

Answer 2

I would suggest you try replacing the lines :

<!-- Set up the AmqpTemplate/RabbitTemplate: -->
<rabbit:connection-factory id="rabbitConnectionFactory"/>

with :

<!-- Obtain a connection to the RabbitMQ via cloudfoundry-runtime: -->
<cloud:rabbit-connection-factory id="connectionFactory"/>

The latter case is using the cloud namespace which will read the rabbit-