PoorbandTony
PoorbandTony

Reputation: 432

ODP.net - ORA-12537 End of File errors when calling stored procedure

I'm perplexed by the above, which keeps cropping up intermittently whenever I execute a stored proc within a package on our new Oracle installation.

The sproc is called from WCF, which returns the following exception :

<ExceptionDetail xmlns="http://schemas.datacontract.org/2004/07/System.ServiceModel" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">  
           <HelpLink i:nil="true"/>  
           <InnerException>  
              <HelpLink i:nil="true"/>  
              <InnerException i:nil="true"/>  
              <Message>ORA-12537: Network Session: End of file</Message>  
              <StackTrace><![CDATA[at OracleInternal.Network.ReaderStream.Read(OraBuf OB)  
   at OracleInternal.TTC.OraBufReader.GetDataFromNetwork()  
   at OracleInternal.TTC.OraBufReader.Read(Boolean bIgnoreData)  
   at OracleInternal.TTC.MarshallingEngine.UnmarshalUB1(Boolean bIgnoreData)  
   at OracleInternal.TTC.TTCExecuteSql.ReceiveExecuteResponse(Accessor[]& defineAccessors, Accessor[] bindAccessors, Boolean bHasReturningParams, SQLMetaData& sqlMetaData, SqlStatementType statementType, Int64 noOfRowsFetchedLastTime, Int32 noOfRowsToFetch, Int32& noOfRowsFetched, Int64& queryId, Int32 longFetchSize, Int64 initialLOBFetchSize, Int64[] scnFromExecution, Boolean bAllInputBinds, Int32 arrayBindCount, DataUnmarshaller& dataUnmarshaller, MarshalBindParameterValueHelper& marshalBindParamsHelper, Int64[]& rowsAffectedByArrayBind, Boolean bDefineDone, Boolean& bMoreThanOneRowAffectedByDmlWithRetClause, List`1& implicitRSList, Boolean bLOBArrayFetchRequired)]]></StackTrace>  
              <Type>OracleInternal.Network.NetworkException</Type>  
           </InnerException>  
           <Message>ORA-12537: Network Session: End of file</Message>  
           <StackTrace><![CDATA[at Oracle.ManagedDataAccess.Client.OracleException.HandleError(OracleTraceLevel level, OracleTraceTag tag, Exception ex, OracleLogicalTransaction oracleLogicalTransaction)  
   at OracleInternal.TTC.TTCExecuteSql.ReceiveExecuteResponse(Accessor[]& defineAccessors, Accessor[] bindAccessors, Boolean bHasReturningParams, SQLMetaData& sqlMetaData, SqlStatementType statementType, Int64 noOfRowsFetchedLastTime, Int32 noOfRowsToFetch, Int32& noOfRowsFetched, Int64& queryId, Int32 longFetchSize, Int64 initialLOBFetchSize, Int64[] scnFromExecution, Boolean bAllInputBinds, Int32 arrayBindCount, DataUnmarshaller& dataUnmarshaller, MarshalBindParameterValueHelper& marshalBindParamsHelper, Int64[]& rowsAffectedByArrayBind, Boolean bDefineDone, Boolean& bMoreThanOneRowAffectedByDmlWithRetClause, List`1& implicitRSList, Boolean bLOBArrayFetchRequired)  
   at OracleInternal.ServiceObjects.OracleDataReaderImpl.FetchMoreRows(Int32 noOfRowsToFetch, Boolean fillReader, Boolean returnPSTypes)  
   at Oracle.ManagedDataAccess.Client.OracleDataReader.Read()  
   at System.Data.Common.DataAdapter.FillLoadDataRow(SchemaMapping mapping)  
   at System.Data.Common.DataAdapter.FillFromReader(DataSet dataset, DataTable datatable, String srcTable, DataReaderContainer dataReader, Int32 startRecord, Int32 maxRecords, DataColumn parentChapterColumn, Object parentChapterValue)  
   at System.Data.Common.DataAdapter.Fill(DataSet dataSet, String srcTable, IDataReader dataReader, Int32 startRecord, Int32 maxRecords)  
   at Oracle.ManagedDataAccess.Client.OracleDataAdapter.Fill(DataSet dataSet, String srcTable, IDataReader dataReader, Int32 startRecord, Int32 maxRecords)  
   at Oracle.ManagedDataAccess.Client.OracleDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)  
   at System.Data.Common.DbDataAdapter.Fill(DataSet dataSet)  
   at ***REDACTED***  
   at ***REDACTED***  
   at ***REDACTED***  
   at SyncInvokeQtyAdjustments_Get(Object , Object[] , Object[] )  
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)  
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)  
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)  
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(MessageRpc& rpc)  
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)  
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)  
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc)  
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc)  
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)  
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc)  
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)]]></StackTrace>  
           <Type>Oracle.ManagedDataAccess.Client.OracleException</Type>  
        </ExceptionDetail>  

There doesn't seem to be a pattern to it either - sometimes it works fine, sometimes it doesn't.

Here's stuff I've tried :

So, I'm at a loss. I turned on ODP tracing - and did notice it seems to be having issues when dealing with the pool. The first time, I got this :

2016-07-26 11:43:55.268765 TID:6   (PRI) (SVC) (ENT) OracleConnectionImpl.EvaluateDbMajorMinorPatchsetVersion()  
2016-07-26 11:43:55.268765 TID:6   (PRI) (SVC) (EXT) OracleConnectionImpl.EvaluateDbMajorMinorPatchsetVersion()  
2016-07-26 11:43:55.268765 TID:6   (PRI) (SVC) (EXT) OracleConnectionImpl.Connect() (oper=open) (aff=n/a) (inst=WMSQA) (affmatch=n/a) (sessid=2540:42190) (F;F;F;WMSQA;N) (pmid=29382239)  
2016-07-26 11:43:55.268765 TID:6   (PRI) (ENT) (CP) PoolManager`3.PutNewPR() (aff=n/a) (inst=WMSQA) (affmatch=n/a) (sessid=2540:42190) (F;F;F;WMSQA;N) (pmid=29382239)  
2016-07-26 11:43:55.268765 TID:6   (PRI) (ENT) (CP) OraclePool.PutNewPR()  
2016-07-26 11:43:55.268765 TID:6   (PRI) (ENT) (CP) Pool`3.PutNewPR()  
2016-07-26 11:43:55.268765 TID:6   (PRI) (CP) Pool`3.PutNewPR() (oper=cp:put:reg1) (aff=n/a) (inst=WMSQA) (affmatch=n/a) (sessid=2540:42190) (F;F;T;WMSQA;N) (pmid=29382239)  
2016-07-26 11:43:55.268765 TID:6   (PRI) (EXT) (CP) Pool`3.PutNewPR()  
2016-07-26 11:43:55.268765 TID:6   (PRI) (EXT) (CP) OraclePool.PutNewPR()  
2016-07-26 11:43:55.268765 TID:6   (PRI) (EXT) (CP) PoolManager`3.PutNewPR() (aff=n/a) (inst=WMSQA) (affmatch=n/a) (sessid=2540:42190) (F;F;T;WMSQA;N) (pmid=29382239)  
2016-07-26 11:43:55.269765 TID:6   (PRI) (EXT) (CP) PoolManager`3.CreateNewPR() (aff=n/a) (inst=WMSQA) (affmatch=n/a) (sessid=2540:42190) (F;F;T;WMSQA;N) (pmid=29382239)  
2016-07-26 11:43:55.269765 TID:6   (PRI) (CP) PoolManager`3.PopulatePool() PopulatePoolThreadFunc(created=2; max=25; total=3)  
2016-07-26 11:43:55.269765 TID:6   (PRI) (EXT) (CP) PoolManager`3.PopulatePool()  
2016-07-26 11:43:55.880826 TID:15  (PRI) (ENT) TTCExecuteSql.ReceiveExecuteResponse()  
2016-07-26 11:43:55.881826 TID:15  (PRI) (ENT) OracleException.ctor()  
2016-07-26 11:43:55.885826 TID:15  (PRI) (ENT) OracleError.ctor()  
2016-07-26 11:43:55.886826 TID:15  (PRI) (EXT) OracleError.ctor()  
2016-07-26 11:43:55.886826 TID:15  (PRI) (EXT) OracleException.ctor()  
2016-07-26 11:43:55.890827 TID:15  (PRI) (TTC) (ERR) TTCExecuteSql.ReceiveExecuteResponse() (txnid=n/a) OracleInternal.Network.NetworkException (0x000030F9): ORA-12537: Network Session: End of file  
   at OracleInternal.Network.ReaderStream.Read(OraBuf OB)  
   at OracleInternal.TTC.OraBufReader.GetDataFromNetwork()  
   at OracleInternal.TTC.OraBufReader.Read(Boolean bIgnoreData)  
   at OracleInternal.TTC.MarshallingEngine.UnmarshalUB1(Boolean bIgnoreData)  
   at OracleInternal.TTC.TTCExecuteSql.ReceiveExecuteResponse(Accessor[]& defineAccessors, Accessor[] bindAccessors, Boolean bHasReturningParams, SQLMetaData& sqlMetaData, SqlStatementType statementType, Int64 noOfRowsFetchedLastTime, Int32 noOfRowsToFetch, Int32& noOfRowsFetched, Int64& queryId, Int32 longFetchSize, Int64 initialLOBFetchSize, Int64[] scnFromExecution, Boolean bAllInputBinds, Int32 arrayBindCount, DataUnmarshaller& dataUnmarshaller, MarshalBindParameterValueHelper& marshalBindParamsHelper, Int64[]& rowsAffectedByArrayBind, Boolean bDefineDone, Boolean& bMoreThanOneRowAffectedByDmlWithRetClause, List`1& implicitRSList, Boolean bLOBArrayFetchRequired)  

Ramping up the tracing to 127, I just got this :

2016-07-26 14:22:11.459348 TID:1   (PRI) (SVC) (ENT) OracleDataReaderImpl.FetchMoreRows()  
2016-07-26 14:22:11.459348 TID:1   (PRI) (SVC) (ENT) OracleConnectionImpl.AddAllPiggyBackRequests()  
2016-07-26 14:22:11.459348 TID:1   (PRI) (TTC) (ENT) TTCClose.Write()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (ENT) TTCFunction.WriteFunctionHeader()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (ENT) TTCMessage.WriteTTCCode()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (EXT) TTCMessage.WriteTTCCode()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (EXT) TTCFunction.WriteFunctionHeader()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (EXT) TTCClose.Write()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (SVC) (EXT) OracleConnectionImpl.AddAllPiggyBackRequests()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (ENT) TTCExecuteSql.SendExecuteRequest()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (ENT) TTCExecuteSql.GetExecuteOptions()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (EXT) TTCExecuteSql.GetExecuteOptions()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (ENT) TTCExecuteSql.WriteOall8Message()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (ENT) TTCFunction.WriteFunctionHeader()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (ENT) TTCMessage.WriteTTCCode()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (EXT) TTCMessage.WriteTTCCode()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (EXT) TTCFunction.WriteFunctionHeader()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (ENT) TTCExecuteSql.WritePisdef()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (EXT) TTCExecuteSql.WritePisdef()  
2016-07-26 14:22:11.460348 TID:1   (PRI) (TTC) (ENT) TTCExecuteSql.WritePisdefData()  
2016-07-26 14:22:11.461348 TID:1   (PRI) (TTC) (EXT) TTCExecuteSql.WritePisdefData()  
2016-07-26 14:22:11.461348 TID:1   (NET) (SND) 00 4C 00 00 06 00 00 00   |.L......|  
2016-07-26 14:22:11.461348 TID:1   (NET) (SND) 00 00                     |..      |  
2016-07-26 14:22:11.461348 TID:1   (NET) (SND) 11 69 00 01 01 01 01 02   |.i......|  
2016-07-26 14:22:11.461348 TID:1   (NET) (SND) 03 5E 00 02 80 40 01 03   |.^...@..|  
2016-07-26 14:22:11.461348 TID:1   (NET) (SND) 00 00 01 01 0D 00 00 00   |........|  
2016-07-26 14:22:11.461348 TID:1   (NET) (SND) 00 01 01 00 00 00 00 00   |........|  
2016-07-26 14:22:11.461348 TID:1   (NET) (SND) 00 00 00 00 00 00 01 00   |........|  
2016-07-26 14:22:11.461348 TID:1   (NET) (SND) 00 00 00 00 00 00 00 00   |........|  
2016-07-26 14:22:11.461348 TID:1   (NET) (SND) 02 01 36 00 00 00 03 A7   |..6.....|  
2016-07-26 14:22:11.461348 TID:1   (NET) (SND) EB C6 00 01 01 00 00 00   |........|  
2016-07-26 14:22:11.461348 TID:1   (NET) (SND) 00 00                     |..      |  
2016-07-26 14:22:11.461348 TID:1   (PRI) (TTC) (EXT) TTCExecuteSql.WriteOall8Message()  
2016-07-26 14:22:11.461348 TID:1   (PRI) (TTC) (EXT) TTCExecuteSql.SendExecuteRequest()  
2016-07-26 14:22:11.461348 TID:1   (PRI) (TTC) (ENT) TTCExecuteSql.ReceiveExecuteResponse()  
2016-07-26 14:22:11.461348 TID:1   (PRI) (BUF) (COBP.GET) (poolid:2) (key:8192) (bufid:8) (count:5) (OraBufReader.GetDataFromNetwork)  
2016-07-26 14:22:12.161348 TID:1   (PRI) (ENT) TTCExecuteSql.ReceiveExecuteResponse()  
2016-07-26 14:22:12.162348 TID:1   (PRI) (ENT) OracleException.ctor()  
2016-07-26 14:22:12.162348 TID:1   (PRI) (ENT) OracleError.ctor()  
2016-07-26 14:22:12.162348 TID:1   (PRI) (EXT) OracleError.ctor()  
2016-07-26 14:22:12.162348 TID:1   (PRI) (EXT) OracleException.ctor()  
2016-07-26 14:22:12.163348 TID:1   (PRI) (TTC) (ERR) TTCExecuteSql.ReceiveExecuteResponse() (txnid=n/a) OracleInternal.Network.NetworkException (0x000030F9): ORA-12537: Network Session: End of file  
   at OracleInternal.Network.ReaderStream.Read(OraBuf OB)  
   at OracleInternal.TTC.OraBufReader.GetDataFromNetwork()  
   at OracleInternal.TTC.OraBufReader.Read(Boolean bIgnoreData)  
   at OracleInternal.TTC.MarshallingEngine.UnmarshalUB1(Boolean bIgnoreData)  
   at OracleInternal.TTC.TTCExecuteSql.ReceiveExecuteResponse(Accessor[]& defineAccessors, Accessor[] bindAccessors, Boolean bHasReturningParams, SQLMetaData& sqlMetaData, SqlStatementType statementType, Int64 noOfRowsFetchedLastTime, Int32 noOfRowsToFetch, Int32& noOfRowsFetched, Int64& queryId, Int32 longFetchSize, Int64 initialLOBFetchSize, Int64[] scnFromExecution, Boolean bAllInputBinds, Int32 arrayBindCount, DataUnmarshaller& dataUnmarshaller, MarshalBindParameterValueHelper& marshalBindParamsHelper, Int64[]& rowsAffectedByArrayBind, Boolean bDefineDone, Boolean& bMoreThanOneRowAffectedByDmlWithRetClause, List`1& implicitRSList, Boolean bLOBArrayFetchRequired)  
2016-07-26 14:22:12.163348 TID:1   (PRI) (EXT) TTCExecuteSql.ReceiveExecuteResponse()  
2016-07-26 14:22:12.163348 TID:1   (PRI) (TTC) (EXT) TTCExecuteSql.ReceiveExecuteResponse()  
2016-07-26 14:22:12.163348 TID:1   (PRI) (ENT) OracleDataReaderImpl.FetchMoreRows()  
2016-07-26 14:22:12.163348 TID:1   (PRI) (SVC) (ERR) OracleDataReaderImpl.FetchMoreRows() (txnid=n/a) Oracle.ManagedDataAccess.Client.OracleException (0x80004005): ORA-12537: Network Session: End of file ---> OracleInternal.Network.NetworkException (0x000030F9): ORA-12537: Network Session: End of file  

I tweaked the query in the sproc by commenting out the select statements and where clauses, and gradually introduced them back in to see if it's the query that's the problem (running it in Oracle Developer works fine, though). I've narrowed it down to this statement - if I reintroduce it, I get the End of file exceptions again :

WHERE  
      -- Only handle records that contain a numeric value in the ref field.  
      ISNUMERIC(PTT.REF_FIELD_2) = 1  

This is what the ISNUMERIC function looks like in the package ;

  FUNCTION ISNUMERIC (p_string IN VARCHAR2) RETURN INT  
  IS  
    v_new_num NUMBER;  
  BEGIN  
    IF p_string IS NULL  
    THEN  
      RETURN 0;  
    END IF;  

    v_new_num := TO_NUMBER(p_string);  
    RETURN 1;  
  EXCEPTION  
    WHEN VALUE_ERROR THEN  
      RETURN 0;  
  END ISNUMERIC;  

At first I thought it might be the number of exceptions caught was preventing further execution due to the number of exceptions. However I simplified the function to just return '1' and it still didn't work every time.

The results of the query are passed back via a ref cursor e.g.

PROCEDURE DEVSP_API_GETxxxx(  
    P_STARTDATE IN DATE  
  , P_ENDDATE IN DATE   
  , P_xxx IN P_xxx_TYPE   
  , <snip>  

  , P_DATAOUT OUT sys_refcursor  
  ) AS  
  BEGIN  

    OPEN P_DATAOUT FOR  


    SELECT   
      'asdsa' as "x",  
      'sadasdsad' as "z",  
      0 as "Qty",  
<snip>  

Finally, I removed all Function calls from the select statements and the where clauses - and that instantly stops the error from occurring - I replaced them with the same logic the function would run - just inline within the SELECT and WHERE statements.

So what's going on? Is there a limitation in ODP.net when running sprocs within a package that contain calls to functions? Seems a bit odd.

The workaround is to just not use functions, but I don't want to do that as some select statements have conditional expressions in them which repeat the same logic in Union-based queries.

Any help or pointers much appreciated!

Ps. for transparency, I posted this on the Oracle forum last week but have had no responses.

Update : 03-Aug-2016

Finally managed to get access to the server to try a bit of tracing. Here's what came up in the trace log when the process ends prematurely :

*** 2016-08-03 10:14:39.111
*** SESSION ID:(2526.53330) 2016-08-03 10:14:39.111
*** CLIENT ID:() 2016-08-03 10:14:39.111
*** SERVICE NAME:(XXXXX) 2016-08-03 10:14:39.111
*** MODULE NAME:(iisexpress.exe) 2016-08-03 10:14:39.111
*** CLIENT DRIVER:(ODPM.NET) 2016-08-03 10:14:39.111
*** ACTION NAME:() 2016-08-03 10:14:39.111

Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x3CEBE1D, pmucfst()+13] [flags: 0x0, count: 1]
Incident 109041 created, dump file /u01/app/oracle/diag/rdbms/xxxx/XXXX/incident/incdir_109041/XXXXX_ora_13149_i109041.trc
ORA-07445: exception encountered: core dump [pmucfst()+13] [SIGSEGV] [ADDR:0x0] [PC:0x3CEBE1D] [Address not mapped to object] []

ssexhd: crashing the process...
Shadow_Core_Dump = PARTIAL

I can also see a core dump in the same folder. I've tried grepping the bucket trace files in there to see if there's anything obvious - but to be honest it's difficult to search for something when you don't know what you're looking for!

I can see a shedload of core dumps from previous attempts to run the sprocs whenever it used a function - so it's at least reasonably consistent.

Upvotes: 1

Views: 4141

Answers (1)

Brad Powell
Brad Powell

Reputation: 11

I ran into the same issue. Oracle Support responded with:

It looks like you are hitting Bug 18191823 - Hang / ORA-600 [19708] etc.. referencing stale REFCURSOR bind ( Doc ID 18191823.8 )

Workaround
Ensure the duration of the bound PL/SQL variable is at least as long as that of the ref cursor to which it is bound.

The fix is first included in July 2016 patches.

Upvotes: 1

Related Questions