CODEWITHSUNDEEP
linuxshellfilescripting
lord_ozb
lord_ozb

Reputation: 95

Faster way to split way big file in to smaller files?

I have a small file which is about 6.5 GB and I tried to split it into files of size 5MB each using split -d -line--bytes=5MB. It took me over 6 minutes to split this file. I have files over 1TB. Is there a faster way to do this?

Upvotes: 2

Views: 410

Answers (1)

l0b0
l0b0

Reputation: 58808

Faster than a tool specifically designed to do this kind of job? Doesn't sound likely in the general case. However, there are a few things you may be able to do:

  • Save the output files to a different physical storage unit. This avoids reading and writing data to the same disk at the same time, allowing more uninterrupted processing.
  • If the record size is static you can use --bytes to avoid the processing overhead of dealing with full lines.

Upvotes: 3

Related Questions