Error Received in WSO2 MI 7.0.0 and server shutting down automatically

Question

Receiving the error as shared below. Can anyone guide me with solution

Thanks in advance!

[2021-06-14 07:21:00,047] ERROR {​​​​​​​org.apache.synapse.mediators.builtin.LogMediator}​​​​​​​ - ERROR_MESSAGE = Error while performing the call operation, ERROR_CODE = 0, ERROR_DETAIL = org.apache.synapse.SynapseException: Error while performing the call operation at org.apache.synapse.mediators.builtin.CallMediator.handleFault(CallMediator.java:432) at org.apache.synapse.mediators.builtin.CallMediator.handleNonBlockingCall(CallMediator.java:281) at org.apache.synapse.mediators.builtin.CallMediator.mediate(CallMediator.java:122) at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:109) at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:71) at org.apache.synapse.mediators.template.TemplateMediator.mediate(TemplateMediator.java:133) at org.apache.synapse.mediators.template.InvokeMediator.mediate(InvokeMediator.java:148) at org.apache.synapse.mediators.template.InvokeMediator.mediate(InvokeMediator.java:84) at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:109) at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:71) at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158) at org.apache.synapse.mediators.MediatorWorker.run(MediatorWorker.java:86) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.synapse.SynapseException: Unexpected error during sending message out at org.apache.synapse.core.axis2.Axis2Sender.handleException(Axis2Sender.java:281) at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:91) at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:571) at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:408) at org.apache.synapse.endpoints.HTTPEndpoint.send(HTTPEndpoint.java:96) at org.apache.synapse.mediators.builtin.CallMediator.handleNonBlockingCall(CallMediator.java:278) ... 13 more Caused by: java.lang.IllegalStateException: I/O reactor has been shut down at org.apache.http.util.Asserts.check(Asserts.java:34) at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:225) at org.apache.synapse.transport.passthru.connections.TargetConnections.getConnection(TargetConnections.java:97) at org.apache.synapse.transport.passthru.DeliveryAgent.submit(DeliveryAgent.java:162) at org.apache.synapse.transport.passthru.PassThroughHttpSender.sendRequestContent(PassThroughHttpSender.java:407) at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:277) at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442) at org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.send(DynamicAxisOperation.java:185) at org.apache.synapse.core.axis2.DynamicAxisOperation$DynamicOperationClient.executeImpl(DynamicAxisOperation.java:167) at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149) at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:650) at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:85) ... 17 more

Answer 1

According to the above error trace the I/O reactor has been shut down. When you try to send messages using an already shut down I/O reactor, you will get an error similar to the above error trace. When the I/O reactor is shut down, there is no way to recover without restarting the MI server. Shutting down the I/O reactor is a critical issue and this usually does not occur due to the configurations, rather it causes due to some environmental factors.

This can be happened due to the insufficient number of open files in the system. Therefore check whether you have followed the steps given in this documentation to optimize the OS level performances?

Further, you can check the allowed file limits for the current WSO2 process using the below command.

Please replace with wso2 process ID.

cat /proc/{PID}/limits

You can check the soft and hard file limits for the user which you use to run the WSO2 EI process using the below commands.

ulimit -Hn
ulimit -Sn