CODEWITHSUNDEEP
coptimizationio
Saurabh
Saurabh

Reputation: 353

Optimize I/O to a large file in C

I have written a C program that reads data from a huge file(>3 GB). Each record in the file is a key-value pair. Whenever a query comes, the program searches for the key and retrieves the corresponding value, similarly for updating the value.
The queries are coming at a fast rate so this technique will eventually fail. The worst case access time is too large. Creating an in-memory object will again be a bad idea, because of the size.
Is there any way in which this problem can be sorted out?

Upvotes: 1

Views: 339

Answers (2)

AJMansfield
AJMansfield

Reputation: 4129

How large are the keys, in comparison to their corresponding values? If they are significantly smaller, you might try creating a table in memory between the keys and the corresponding locations within the file of their values.

Upvotes: 3

David W
David W

Reputation: 10184

Sure seems to me a file of that size wrapping a series of name-value pairs is begging to be migrated to an actual database; failing that, I'd probably at least explore the idea of a memory-mapped file, with only portions resident at any given time...

Upvotes: 5

Related Questions