CODEWITHSUNDEEP
c++file-io
user3697365
user3697365

Reputation: 61

Big File reading error in C++

I need to read a file in c++ that has this specific format:

10 5
1 2 3 4 1 5 1 5 2 1

All the values are separated with a space. The first 2 on the first line are the variables N and M respectively and all the N values from the second line need to be in an array called S with the size of N. The code I have written has no problem with files like these but it does not work when it comes to really big files with millions and so on that i need it to work with. Here is the code

int N,M;
FILE *read = fopen("file.in", "r");
fscanf(read, "%d %d ", &N, &M);
int S[N];
for( i =0; i < N; i++){
    fscanf(read, "%d ", &S[i]);        
}

What should I change?

Upvotes: 1

Views: 117

Answers (2)

Matthieu M.
Matthieu M.

Reputation: 299800

There are multiple potential issues when getting in the range of millions of integers:

  • int is most often 32 bits, a 32 bits signed integer will have a range of -2^31 to 2^31 - 1, and thus the maximum of 2,147,483,647. You should switch to a 64 bits integral.

  • You are using int S[N] a Variable Length Array (VLA) which is not Standard C++ (it is Standard C99, but... there are discussions as to whether it was a good idea or not). The important detail, though, is that a VLA is stored on the stack: 1 million of 32 bits int is 4 MB, 2 millions is 8 MB, etc... check your default stack size, but it likely is less than 8 MB, and thus you have a stack-overflow (you're on the right site for help!).

So, let's switch to C++ and do away with those issues:

#include <cstdint> // for int64_t
#include <fstream>
#include <vector>

int main(int argc, char* argv[]) {
   std::ifstream stream("data.txt");

   int64_t n = 0, m = 0;
   stream >> n >> m;

   std::vector<int> data;
   for (int64_t c = 0; c != n; ++c) {
       int i = 0;
       stream >> i;
       data.push_back(i);
   }

   // do your best :)
}

First of all, we use int64_t from <cstdint> to do away with the integer overflow issue. Second, we use a stream (input file stream: ifstream) to avoid having to learn what is the format associated with each and every integral type (it's a pain). Third, we use a vector to store the data we read, and do away with the stack overflow issue.

Upvotes: 1

Christophe
Christophe

Reputation: 73376

You are using variable sized arrays. This is not standard and not supported by all compilers. If your compiler support it, and you go in the millions, you'll run out of stack space (stack overflow).

Alternatively, you could define S as being a vector with vector<int> S(N);

Upvotes: 1

Related Questions