CODEWITHSUNDEEP
c#sql-serveroracle-databaseetl
Shawn
Shawn

Reputation: 5260

C# SqlBulkCopy resume on error

I am trying to do a bulk insert of 10 million of rows from Oracle to SQL Server using BulkCopy.WriteToServer().

I have made sure

When it came to just about 1.4 million rows it failed with System.ArgumentOutOfRangeException: Hour, Minute, and Second parameters describe an un-representable DateTime. at System.DateTime.DateToTicks(Int32 year, Int32 month, Int32 day)

Here is my code

      SqlBulkCopy copy;
      copy = new SqlBulkCopy(destConn, SqlBulkCopyOptions.TableLock, null); 
      // ColumnMappings property is used to map column positions, not data type
      copy.DestinationTableName = DestTable;
      copy.NotifyAfter = 5000; 
      copy.SqlRowsCopied += new SqlRowsCopiedEventHandler(OnSqlRowsCopied);
      copy.BulkCopyTimeout = 0;
      try { copy.WriteToServer((IDataReader)rd); }
      catch (Exception ex)
      {
        AppInfo.TableMsg[SrcTable] = AppInfo.TableMsg[SrcTable] + "\r\n" + "bulkcopy.WriteToServer(rd) failed. " + ex.Message;
        throw ex;
      }

My table got over 100 columns and there are 26 DATE columns. It's hard to sort out where the bad data is

So I got 3 questions here

  1. Is there any setting/option to make WriteToServer() continue or ignore exception? Or any way I can do in catch block to make it continue? I don't care leaving the bad data behind. I am looking for a way to tell it to continue on insertion errors.
  2. Is there any way to prevent this from happening? For example anything I can to in select query that fills the OracleDataReader?
  3. If no solution to the above 2 questions, then is there any good way to cleanse the "bad date" out on Oracle side?

Thanks,

Update: I have done the following

  1. Change destination table data type from datetime to datetime2

  2. Modify the select list to

    CASE WHEN my_date_column < To_Date('01/01/1753', 'mm/dd/yyyy') THEN To_Date('01/01/1753','mm/dd/yyyy') ELSE my_date_column END

    for all the columns with DATE datatype.

But the error still persists. Here is complete error message.

System.ArgumentOutOfRangeException was caught
  HResult=-2146233086
  Message=Hour, Minute, and Second parameters describe an un-representable DateTime.
  Source=mscorlib
  StackTrace:
       at System.DateTime.TimeToTicks(Int32 hour, Int32 minute, Int32 second)
       at Oracle.DataAccess.Client.OracleDataReader.GetDateTime(Int32 i)
       at Oracle.DataAccess.Client.OracleDataReader.GetValue(Int32 i)
       at System.Data.SqlClient.SqlBulkCopy.GetValueFromSourceRow(Int32 destRowIndex, Boolean& isSqlType, Boolean& isDataFeed, Boolean& isNull)
       at System.Data.SqlClient.SqlBulkCopy.ReadWriteColumnValueAsync(Int32 col)
       at System.Data.SqlClient.SqlBulkCopy.CopyColumnsAsync(Int32 col, TaskCompletionSource`1 source)
       at System.Data.SqlClient.SqlBulkCopy.CopyRowsAsync(Int32 rowsSoFar, Int32 totalRows, CancellationToken cts, TaskCompletionSource`1 source)
       at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsyncContinued(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
       at System.Data.SqlClient.SqlBulkCopy.CopyBatchesAsync(BulkCopySimpleResultSet internalResults, String updateBulkCommandText, CancellationToken cts, TaskCompletionSource`1 source)
       at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestContinuedAsync(BulkCopySimpleResultSet internalResults, CancellationToken cts, TaskCompletionSource`1 source)
       at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalRestAsync(CancellationToken cts, TaskCompletionSource`1 source)
       at System.Data.SqlClient.SqlBulkCopy.WriteToServerInternalAsync(CancellationToken ctoken)
       at System.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServerAsync(Int32 columnCount, CancellationToken ctoken)
       at System.Data.SqlClient.SqlBulkCopy.WriteToServer(IDataReader reader)

From the error message it looks like the offending part is OracleDataReader rather than SqlBulkCopy.

How can I quickly spot these offending value using a Oracle query? Any further suggestions?

Upvotes: 1

Views: 2391

Answers (3)

Jana Sattainathan
Jana Sattainathan

Reputation: 825

The problem is well described by "usr"'s answer. You can do a CASE statement on the source Oracle source columns with the problem dates to convert invalid dates to NULL or a default value. Now, identifying the rows or columns with the problem is a BIG issue. I have comeup with a method to identify the problem.

Please read this blog post of mine to identify the problem columns so that you can do the appropriate DECODE to convert the problem dates to NULL or valid defaults https://sqljana.wordpress.com/tag/datetime-odp-net-oracle/

Upvotes: 0

Shawn
Shawn

Reputation: 5260

OK. I figure it out. I am answering my 2nd and 3rd questions.

It is the bad date in Oracle that looks like this '01/26/2006 17:94:00'.

To_char(my_column,'hh24:mi:ss') shows '00:00:00' To_char(my_column,'mi') shows '00'

It appear as valid data and cannot be identified as invalid by using to_char() function as filter

What I can do is using dump function

DELETE FROM my_table
WHERE my_column IS NOT NULL
  AND (To_Number(SubStr(Dump(my_column), InStr(Dump(my_column),':',1,1)+2, InStr(Dump(my_column),',',1,1)-InStr(Dump(my_column),':',1,1)-2))-100 < 0
   OR To_Number(SubStr(Dump(my_column), InStr(Dump(my_column),',',1,2)+1, InStr(Dump(my_column),',',1,3)-1-InStr(Dump(my_column),',',1,2))) NOT BETWEEN 1 AND 12
   OR To_Number(SubStr(Dump(my_column), InStr(Dump(my_column),',',1,3)+1, InStr(Dump(my_column),',',1,4)-1-InStr(Dump(my_column),',',1,3))) NOT BETWEEN 1 AND 31
   OR To_Number(SubStr(Dump(my_column), InStr(Dump(my_column),',',1,4)+1, InStr(Dump(my_column),',',1,5)-1-InStr(Dump(my_column),',',1,4))) NOT BETWEEN 1 AND 24
   OR To_Number(SubStr(Dump(my_column), InStr(Dump(my_column),',',1,5)+1, InStr(Dump(my_column),',',1,6)-1-InStr(Dump(my_column),',',1,5))) NOT BETWEEN 1 AND 60
   OR To_Number(SubStr(Dump(my_column), InStr(Dump(my_column),',',-1)+1)) NOT BETWEEN 1 AND 60)

And that cleans up the bad data.

Upvotes: 0

usr
usr

Reputation: 171206

Oracle Database can store dates in the Julian era, ranging from January 1, 4712 BCE through December 31, 9999 CE (Common Era, or 'AD'). Unless BCE ('BC' in the format mask) is specifically used, CE date entries are the default.

SQL Server's datetime cannot do that. datetime2 is recommended for new development and it can hold all practical date and time values. If you still hit any range limit, run Oracle queries of the style SELECT * FROM T WHERE SomeDateCol < '0000-01-01' to find the invalid data.

TL;DR: Research the exact supported value ranges and find any values that cannot be mapped.

Your questions:

  1. No, SQL Server cannot do that alas.
  2. Yes, treat invalid rows differently. Maybe filter them out or convert the invalid values to NULL. Your choice.
  3. See above.

Upvotes: 1

Related Questions