CODEWITHSUNDEEP
c++file-io
Schemer
Schemer

Reputation: 1665

C++ Optimal Block Size For Reading From A File

I have a program that generates files containing random distributions of the character A - Z. I have written a method that reads these files (and counts each character) using fread with different buffer sizes in an attempt to determine the optimal block size for reads. Here is the method:

int get_histogram(FILE * fp, long *hist, int block_size, long *milliseconds, long *filelen)
{
    char *buffer = new char[block_size];
    bzero(buffer, block_size);

    struct timeb t;
    ftime(&t);
    long start_in_ms = t.time * 1000 + t.millitm;

    size_t bytes_read = 0;
    while (!feof(fp))
    {
        bytes_read += fread(buffer, 1, block_size, fp);
        if (ferror (fp))
        {
            return -1;
        }
        int i;
        for (i = 0; i < block_size; i++)
        {
            int j;
            for (j = 0; j < 26; j++)
            {
                if (buffer[i] == 'A' + j)
                {
                    hist[j]++;
                }
            }
        }
    }

    ftime(&t);
    long end_in_ms = t.time * 1000 + t.millitm;
    *milliseconds = end_in_ms - start_in_ms;
    *filelen = bytes_read;

    return 0;
}

However, when I plot bytes/second vs. block size (buffer size) using block sizes of 2 - 2^20, I get an optimal block size of 4 bytes -- which just can't be correct. Something must be wrong with my code but I can't find it.

Any advice is appreciated.

Regards.

EDIT:

The point of this exercise is to demonstrate the optimal buffer size by recording the read times (plus computation time) for different buffer sizes. The file pointer is opened and closed by the calling code.

Upvotes: 0

Views: 2287

Answers (2)

unwind
unwind

Reputation: 399863

There are many bugs in this code:

  • It uses new[], which is C++.
  • It doesn't free the allocated memory.
  • It always loops over block_size bytes of input, not bytes_read as returned by fread().

Also, the actual histogram code is rather inefficient, since it seems to loop over each character to determine which character it is.

UPDATE: Removed claim that using feof() before I/O is wrong, since that wasn't true. Thanks to Eric for pointing this out in a comment.

Upvotes: 2

class stacker
class stacker

Reputation: 5347

You're not stating what platform you're running this on, and what compile time parameters you use.

Of course, the fread() involves some overhead, leaving user mode and returning. On the other hand, instead of setting the hist[] information directly, you're looping through the alphabet. This is unnecessary and, without optimization, causes some overhead per byte.

I'd re-test this with hist[j-26]++ or something similar.

Typically, the best timing would be achieved if your buffer size equals the system's buffer size for the given media.

Upvotes: 0

Related Questions