I have data that is streamed from disk and processed in memory by a Java application and that finally needs to be copied into SQL Server. The data can be fairly large (hence the streaming) and can require up to several 100,000 rows to be inserted. The fastest solution seems to be using SQL Server's bulk-copy feature. However, I haven't found any way for Java programs to do this easily or nearly fast enough. Here are some ways that I've already investigated: Using the SqlBulkCopy class in .NET. This is very efficient since you can stream data right from a data source and straight to SQL Server. The problem with this approach is that you need to be running .NET. Perhaps this could be used using a Java to .NET bridge. Although, I wonder about the cost of marshalling data between runtimes. Using the BULK INSERT TSQL statement. The problem with this is that you need create a properly formatted file on disk. I've seen some small performance gains over JDBC's batch insert using this. Also, this is only useful locally. Write files to disk and use the bcp command line utility. Still a little faster than JDBC batch insert but not that much. I also lose the ability to use a transaction with this method. Use the C API . Again, very efficient, but you need to be using C. There would be a way to use this through JNI. If there's some free Java library out there that does this, I'd like to know about it. I'm looking for the fastest solution. Memory is not an issue. Thanks!

Reputation: 143

What's the most efficient way to bulk-copy to SQL Server from Java?

I have data that is streamed from disk and processed in memory by a Java application and that finally needs to be copied into SQL Server. The data can be fairly large (hence the streaming) and can require up to several 100,000 rows to be inserted. The fastest solution seems to be using SQL Server's bulk-copy feature. However, I haven't found any way for Java programs to do this easily or nearly fast enough.

Here are some ways that I've already investigated:

Using the SqlBulkCopy class in .NET. This is very efficient since you can stream data right from a data source and straight to SQL Server. The problem with this approach is that you need to be running .NET. Perhaps this could be used using a Java to .NET bridge. Although, I wonder about the cost of marshalling data between runtimes.
Using the BULK INSERT TSQL statement. The problem with this is that you need create a properly formatted file on disk. I've seen some small performance gains over JDBC's batch insert using this. Also, this is only useful locally.
Write files to disk and use the bcp command line utility. Still a little faster than JDBC batch insert but not that much. I also lose the ability to use a transaction with this method.
Use the C API. Again, very efficient, but you need to be using C. There would be a way to use this through JNI. If there's some free Java library out there that does this, I'd like to know about it.

I'm looking for the fastest solution. Memory is not an issue.

Thanks!

Upvotes: 8

Answers (3)

Christian d'Heureuse

Reputation: 5630

Since version 4.2 of the Microsoft JDBC driver for SQL Server, there is a class named com.microsoft.sqlserver.jdbc.SQLServerBulkCopy which does the same as the SqlBulkCopy class of .NET.

Upvotes: 1

SAsInSumit

Reputation: 21

The best option for me was to use the commercial SQL Server JDBC driver from DataDirect with standard JDBC calls addBatch/executeBatch that run across Linux and Windows - https://blogs.datadirect.com/2012/05/how-to-bulk-insert-jdbc-batches-into-microsoft-sql-server-oracle-sybase.html

I've seen load times improve from 7 hours to under 30 minutes.

Upvotes: 0

Horcrux7

Reputation: 24457

For the .NET answer i would recommended IKVM. Then your Java Code will be .NET code and you can call any .NET code.
The BULK INSERT required also that the bulk file is accessible from SQL Server. This is only a local option. The performance from Batch Update can be vary between different JDBC drivers.
For native calls I would recommended to use JNA (Java native access). Then you does not need to write any C code.

Upvotes: 1

What&#39;s the most efficient way to bulk-copy to SQL Server from Java?

Answers (3)

Related Questions

What's the most efficient way to bulk-copy to SQL Server from Java?