CODEWITHSUNDEEP
cross-platformfilesystemstext-filesbulkinsert
Jason Baker
Jason Baker

Reputation: 198597

What's a reasonable maximum size that a cross-platform application should allow a text file to get?

What's a reasonable maximum size that a cross-platform application could allow a text file to get? I understand that this is an oversimplified question so allow me to explain.

My team is implementing a bulk load interface for clients to load data into our database. It will write out a CSV file and then load that file into the appropriate database (at this point either Oracle or SQL Server). We could be dealing with a relatively high number of records.

Is there any limit I should put on the size of these text files before I start breaking it up into multiple text files? Currently, we're deploying to Linux and Windows, but we also have developers using OS X. Plus, some of our clients have somewhat dated versions of these operating systems. I'd imagine that this is dependent upon the OS, file system, and RDBMS that we're connecting to. Rather than trying to set a limit for each individual platform, I'd like to just have one overall limit for simplicity's sake (as long as that limit isn't overly restrictive). Is this even necessary, or is there a cap I can set across the board?

Upvotes: 1

Views: 434

Answers (1)

Joachim Sauer
Joachim Sauer

Reputation: 308051

Most modern systems have no problem handling multi-gigabyte files, but if you want to be cautious, then setting a limit of 2GB can be useful:

  • Even slightly out-of-date file systems have no problem storing 2GB files (for example FAT16)
  • 2GB can be addressed by a signed 32-bit integer, which is used more often than one might think

For the filesystem part this comparison of file systems might be useful (it lists a lot of not-really-widely-used systems as well, 'though).

Upvotes: 7

Related Questions