CODEWITHSUNDEEP
cfile-io
shunyo
shunyo

Reputation: 1307

Writing large binary file part by part

I am trying to do a fairly easy task but am stumped by how the FILE* works. So I am writing a binary file (las file) which has a header and then the points in binary format. Since this is a huge file, I am doing it part by part. But the problem comes when the file pointer writes almost 3/4th of the file correctly and then gives entirely wrong file position pointer. 

//struct points written to 
struct las_points{
    float x;
    float y;
    float z;
    float time;
};
int writepoints(){
     outfilename = "pointcloud_13.las";
     fout = fopen(outfilename,"wb+");
     // numPts - total number of points to be processed
     // numRec - number of records processed every time
     numRec = numPts/4;
     las_points *lp;
     // ending size of the file
     int endSize = 0,j=0;
     // header is written 
     // endSize = 233
     for(i=0;i<numPts;i+=numRec){
        lp = (las_points*)malloc(sizeof(las_points)*numRec);
        // some processing done 
        printf("Iteration - %d\n",j);
        lasWritePoints(lp,fout,numRec,endSize);
        fflush(fout);
        free(lp);
        lp = NULL;
     }
     fclose(fout);
}
int lasWritePoints(las_points*lp, FILE* fout,int numRec,int &endSize){
     if(!fout || !lp){
          printf("File couldnt be written \n");
          exit(1);
     }
     printf("endSize - %d \n",endSize);
     fseek(fout,endSize,SEEK_SET);
     for(int i=0;i<numRec;i++) {
         fwrite(&lp[i].x,sizeof(float),1,fout);
         fwrite(&lp[i].y,sizeof(float),1,fout);
         fwrite(&lp[i].z,sizeof(float),1,fout);
         fseek(fout,8,SEEK_CUR);
         fwrite(&lp[i].time,sizeof(float),1,fout);
     }
     endSize = ftell(fout);
     printf("endSize - %d \n",endSize);
}

Only the writing of the binary file is reproduced. The problem is that for the first four iterations for a file, it runs smoothly. Then at the end of last iteration, the endSize it gives is lesser than the beginning endSize. 

 Output:
 Total size of file to be read: 1258456752
 Iteration : 0
 endSize : 233
 endSize : 550575041
 Iteration : 1
 endSize : 550575041
 endSize : 1101149849
 Iteration : 2
 endSize : 1101149849
 endSize : 1651724657
 Iteration : 3
 endSize : 1651724657
 endSize : 54815783

Can someone point out what I am doing wrong?

Upvotes: 0

Views: 90

Answers (3)

Bob Blogge
Bob Blogge

Reputation: 188

You need to dereference endSize.

You are assigning the file size/position to a pointer resulting in your unexpected behavior.

endSize = ftell(fout);

s/b

*endSize = ftell(fout);

This is one of the most common mistakes in C.

Upvotes: 0

LogicG8
LogicG8

Reputation: 1757

You are probably getting stuck with the 32-bit versions of file access functions. What platform/compiler/file system are you using?

If you are on:

  • Linux using gcc try adding -D_FILE_OFFSET_BITS=64 to your compiler command
  • Windows using mingw try using fseeko and ftello
  • a 32-bit Windows with Visual C++ try using _open (not quite a drop-in replacement)
  • a file system that has a 2GB file size limit then my condolences

Upvotes: 0

user149341
user149341

Reputation:

You are writing more bytes than can be represented by a 32-bit int (about 2 billion = 2 GB). Use a long to store the results of ftell():

long endSize = ftell(fout);
printf("endSize - %ld\n", endSize);

Upvotes: 1

Related Questions