CODEWITHSUNDEEP
ibm-mq
jonathan white
jonathan white

Reputation: 33

Failed to put message to the backout queue Message broker

I have developed a flow which picks up a message from MQInput, Inserts it into the DB by calling a stored procedure and then dumps the message to another queue. The message is in XMLNSC.

The flow was in running state for 2 months but now I am getting exceptions of weird issues.

3 days back I got this kind of exception:

    (0x01000000:Name):RecoverableException = (
        (0x03000000:NameValue):File                 = '/build/slot1/S900_P/src/DataFlowEngine/MessageServices/ImbDataFlowNode.cpp' (CHARACTER)
        (0x03000000:NameValue):Line                 = 1153 (INTEGER)
        (0x03000000:NameValue):Function             = 'ImbDataFlowNode::createExceptionList' (CHARACTER)
        (0x03000000:NameValue):Type                 = 'ComIbmComputeNode' (CHARACTER)
        (0x03000000:NameValue):Name                 = 'SiebelBOQManagement#FCMComposite_1_2' (CHARACTER)
        (0x03000000:NameValue):Label                = 'SiebelBOQManagement.Compute' (CHARACTER)
        (0x03000000:NameValue):Catalog              = 'BIPmsgs' (CHARACTER)
        (0x03000000:NameValue):Severity             = 3 (INTEGER)
        (0x03000000:NameValue):Number               = 2230 (INTEGER)
        (0x03000000:NameValue):Text                 = 'Node throwing exception' (CHARACTER)
        (0x01000000:Name     ):Insert               = (
          (0x03000000:NameValue):Type = 14 (INTEGER)
          (0x03000000:NameValue):Text = 'SiebelBOQManagement.Compute' (CHARACTER)
        )
        (0x01000000:Name     ):RecoverableException = (
          (0x03000000:NameValue):File                 = '/build/slot1/S900_P/src/DataFlowEngine/ImbRdl/ImbRdlStatementGroup.cpp' (CHARACTER)
          (0x03000000:NameValue):Line                 = 666 (INTEGER)
          (0x03000000:NameValue):Function             = 'SqlStatementGroup::execute' (CHARACTER)
          (0x03000000:NameValue):Type                 = 'ImbESQLManager' (CHARACTER)
          (0x03000000:NameValue):Name                 = 'ImbESQLManager' (CHARACTER)
          (0x03000000:NameValue):Label                = 'ImbESQLManager' (CHARACTER)
          (0x03000000:NameValue):Catalog              = 'BIPmsgs' (CHARACTER)
          (0x03000000:NameValue):Severity             = 3 (INTEGER)
          (0x03000000:NameValue):Number               = 2488 (INTEGER)
          (0x03000000:NameValue):Text                 = 'Error detected, rethrowing' (CHARACTER)
          (0x01000000:Name     ):Insert               = (
            (0x03000000:NameValue):Type = 5 (INTEGER)
            (0x03000000:NameValue):Text = '.SiebelBOQManagement_Compute.Main' (CHARACTER)
          )
          (0x01000000:Name     ):Insert               = (
            (0x03000000:NameValue):Type = 5 (INTEGER)
            (0x03000000:NameValue):Text = '20.3' (CHARACTER)
          )
          (0x01000000:Name     ):Insert               = (
            (0x03000000:NameValue):Type = 5 (INTEGER)
            (0x03000000:NameValue):Text = 'LOGDEBUG(CAST(startTime AS CHARACTER) || '|SiebelBOQManagement|Request|' || CAST(ASBITSTREAM(InputRoot.XMLNSC) AS CHARACTER CCSID 1208), 'Siebel');' (CHARACTER)
          )
          (0x01000000:Name     ):RecoverableException = (
            (0x03000000:NameValue):File                 = '/build/slot1/S900_P/src/DataFlowEngine/ImbRdl/ImbRdlRoutine.cpp' (CHARACTER)
            (0x03000000:NameValue):Line                 = 1450 (INTEGER)
            (0x03000000:NameValue):Function             = 'SqlRoutine::invoke' (CHARACTER)
            (0x03000000:NameValue):Type                 = 'ImbESQLManager' (CHARACTER)
            (0x03000000:NameValue):Name                 = 'ImbESQLManager' (CHARACTER)
            (0x03000000:NameValue):Label                = 'ImbESQLManager' (CHARACTER)
            (0x03000000:NameValue):Catalog              = 'BIPmsgs' (CHARACTER)
            (0x03000000:NameValue):Severity             = 3 (INTEGER)
            (0x03000000:NameValue):Number               = 2934 (INTEGER)
            (0x03000000:NameValue):Text                 = 'Error occured in procedure' (CHARACTER)
            (0x01000000:Name     ):Insert               = (
              (0x03000000:NameValue):Type = 5 (INTEGER)
              (0x03000000:NameValue):Text = 'LOGDEBUG' (CHARACTER)
            )
            (0x01000000:Name     ):RecoverableException = (
              (0x03000000:NameValue):File                 = '/build/slot1/S900_P/src/DataFlowEngine/ImbRdl/ImbRdlExternalJava.cpp' (CHARACTER)
              (0x03000000:NameValue):Line                 = 1458 (INTEGER)
              (0x03000000:NameValue):Function             = 'SqlExternalJavaMethod::executeJavaMethod' (CHARACTER)
              (0x03000000:NameValue):Type                 = '' (CHARACTER)
              (0x03000000:NameValue):Name                 = '' (CHARACTER)
              (0x03000000:NameValue):Label                = '' (CHARACTER)
              (0x03000000:NameValue):Catalog              = 'BIPmsgs' (CHARACTER)
              (0x03000000:NameValue):Severity             = 3 (INTEGER)
              (0x03000000:NameValue):Number               = 2919 (INTEGER)
              (0x03000000:NameValue):Text                 = 'The following error occurred during execution of an Java Method call' (CHARACTER)
              (0x01000000:Name     ):Insert               = (
                (0x03000000:NameValue):Type = 5 (INTEGER)
                (0x03000000:NameValue):Text = '.LOGDEBUG' (CHARACTER)
              )
              (0x01000000:Name     ):Insert               = (
                (0x03000000:NameValue):Type = 5 (INTEGER)
                (0x03000000:NameValue):Text = '1.1' (CHARACTER)
              )
              (0x01000000:Name     ):Insert               = (
                (0x03000000:NameValue):Type = 5 (INTEGER)
                (0x03000000:NameValue):Text = 'com.ufone.utility.ProvLogger.LogDebug' (CHARACTER)
              )
              (0x01000000:Name     ):Insert               = (
                (0x03000000:NameValue):Type = 5 (INTEGER)
                (0x03000000:NameValue):Text = ''1435077650205|SiebelBOQManagement|Request|<Message MessageId="1-7XUQQC" IntObjectName="PTCL Order Entry EAI IO" MessageType="Integration Object" IntObjectFormat="Siebel Hierarchical">.......</Message>
 )
          (0x01000000:Name     ):RecoverableException = (
            (0x03000000:NameValue):File     = '/build/slot1/S900_P/src/DataFlowEngine/ImbRdl/ImbRdlExternalJava.cpp' (CHARACTER)
            (0x03000000:NameValue):Line     = 1339 (INTEGER)
            (0x03000000:NameValue):Function = 'SqlExternalJavaMethod::executeJavaMethod' (CHARACTER)
            (0x03000000:NameValue):Type     = '' (CHARACTER)
            (0x03000000:NameValue):Name     = '' (CHARACTER)
            (0x03000000:NameValue):Label    = '' (CHARACTER)
            (0x03000000:NameValue):Catalog  = 'BIPmsgs' (CHARACTER)
            (0x03000000:NameValue):Severity = 3 (INTEGER)
            (0x03000000:NameValue):Number   = 2917 (INTEGER)
            (0x03000000:NameValue):Text     = 'Java Exception calling CallStaticVoidMethodA.' (CHARACTER)
            (0x01000000:Name     ):Insert   = (
              (0x03000000:NameValue):Type = 5 (INTEGER)
              (0x03000000:NameValue):Text = 'com.ufone.utility.ProvLogger.LogDebug' (CHARACTER)
            )
            (0x01000000:Name     ):Insert   = (
              (0x03000000:NameValue):Type = 5 (INTEGER)
              (0x03000000:NameValue):Text = 'java.lang.OutOfMemoryError: Java heap space' (CHARACTER)
            )
            (0x01000000:Name     ):Insert   = (
              (0x03000000:NameValue):Type = 5 (INTEGER)
              (0x03000000:NameValue):Text = '' (CHARACTER)
            )
          )
        )
      )

When i got this error I knew the memory is full and i stopped the flow. A day later i couldnt find any specific issue in the code and started the flow again.

Mind you that the XML which i am logging is 6000 lines long.

Yesterday I got exceptions in my trace as:

  (0x01000000:Name):RecoverableException = (
(0x03000000:NameValue):File     = '/build/slot1/S900_P/src/DataFlowEngine/MQLibrary/linklib/ImbMqInputNode.cpp' (CHARACTER)
(0x03000000:NameValue):Line     = 2528 (INTEGER)
(0x03000000:NameValue):Function = 'ImbCommonInputNode::eligibleForBackout' (CHARACTER)
(0x03000000:NameValue):Type     = 'ComIbmMQInputNode' (CHARACTER)
(0x03000000:NameValue):Name     = 'SiebelBOQManagement#FCMComposite_1_6' (CHARACTER)
(0x03000000:NameValue):Label    = 'SiebelBOQManagement.MQ Input' (CHARACTER)
(0x03000000:NameValue):Catalog  = 'BIPmsgs' (CHARACTER)
(0x03000000:NameValue):Severity = 3 (INTEGER)
(0x03000000:NameValue):Number   = 2613 (INTEGER)
(0x03000000:NameValue):Text     = 'Failed to put message to the back out queue' (CHARACTER)
(0x01000000:Name     ):Insert   = (
  (0x03000000:NameValue):Type = 5 (INTEGER)
  (0x03000000:NameValue):Text = 'SYSTEM.DEAD.LETTER.QUEUE' (CHARACTER)
)
(0x01000000:Name     ):Insert   = (
  (0x03000000:NameValue):Type = 2 (INTEGER)
  (0x03000000:NameValue):Text = '2' (CHARACTER)
)
(0x01000000:Name     ):Insert   = (
  (0x03000000:NameValue):Type = 2 (INTEGER)
  (0x03000000:NameValue):Text = '2056' (CHARACTER)
)

) )

I am unable to find the root cause. As for a month the Debug procedure was working fine and it wrote the whole XML in the logs and now the flow is giving me abnormal behavior. Also BOQ's are configured for both the input and output queues.

Upvotes: 0

Views: 3490

Answers (1)

AnilReddy
AnilReddy

Reputation: 222

From the above error, there are two possibilities which caused this error.

  1. QMGR file system might be full. Please check /var/mqm file system. If you don't find any issues with this file system, please refere 2nd point
  2. Size of your queue file might be larger than the user file limit size. Please check your file limit size using ulimit -f. And, compare it with your queue file size. I suspect this might be causing you the issue. If so, you can always increase your file system for larger queues using 'ulimit -f 4194302' which is 2GB.

Upvotes: 0

Related Questions