CODEWITHSUNDEEP
python
kaki
kaki

Reputation: 1053

Which is faster?

is opening a large file once reading it completely once to list faster (or) opening smaller files whose total sum of size is equal to large file and loading smaller file into list manupalating one by one faster?

which is faster?? is the difference is time large enough to impact my program?? total time difference of lesser then of 30 sec is negligible for me

Upvotes: 2

Views: 560

Answers (6)

PierreBdR
PierreBdR

Reputation: 43234

It depends if your data fit in your available memory. If you need to resort to paging, or virtual memory, then opening a single giant file might become slower than opening more smaller files. This will be even more true if the computation you need to make creates intermediate variables that won't fit in the physical RAM either.

So, as long as the file is not that big, one opening will be faster, but if this is not true, then many opening may be faster.

At last, note that if you can do many opening, you might be able to do them in parallel and process various parts in different processes, which might make things faster again.

Upvotes: 6

John Machin
John Machin

Reputation: 82924

Why do you think you need to read the file(s) into a list?

If you can open several small files and process each independently, then surely that means:

(a) that you don't need to read into a list, you can process any file (including 1 large file) a line at a time (avoiding running-out-of-real-memory problems)
or
(b) what you need to do is more complicated than you have told us.

Upvotes: 0

nils
nils

Reputation: 628

30sec time difference? Define large. Everything that fits into an average's computer RAM would probably not take much more time than 30sec in total.

Upvotes: 0

Brian R. Bondy
Brian R. Bondy

Reputation: 347216

If you are reading the file sequentially from start until end, one open/close is faster than multiple open/close operations.

However keep in mind that if you need to do a lot of seeking in your 1 big file, then maybe storing separate files won't be slower in that case.

Also keep in mind that no matter which approach you are using, you shouldn't read the entire file in at once. Do it in chunks.

Upvotes: 1

Tomislav Nakic-Alfirevic
Tomislav Nakic-Alfirevic

Reputation: 10173

Working with a single file is almost certainly going to be faster: you have to read the same amount of data in both cases, but when working with multiple files, you have that much more housekeeping operations slowing you down.

Additionally, you can read data from a single file at the maximum speed the disk can handle, using the disk buffer to the maximum etc., whereas with multiple files, the disk head does a lot more dancing jumping from file to file.

Upvotes: 0

dcp
dcp

Reputation: 55424

Obviously one open and close is going to be faster than n opens and closes if you are reading the same amount of data. Plus, when reading a single file the I/O classes you use can take advantage of things like buffering, etc, which makes it even faster.

Upvotes: 2

Related Questions