CODEWITHSUNDEEP
clinuxfilefile-read
SpringUser
SpringUser

Reputation: 45

How to check file completely arrived/copied?

We have a service which process files from a particular folder, whenever files arrived/copied to that folder. Now, before file arrives completely Service is processing and failing with errors. Now, how to check and read only when file completely copied?

The service is running on Linux. After, failing with "ERR_READING", the process will sleep for 5 secs. Even, the process fails to read the file. May be reading the file before completely copied may corrupting the file

Below is the code we are using to read data from file:

int file_read( char *fname, TRANSACTION **tr )
{
    int ret;
    FILE    *fp;

    fp = fopen( fname, "r" );
    if( fp == NULL )
    {
        return( NO_FILE );
    }

    if(( ret = tr_read_tran( fp, tr ))!=0)
        {
        sleep(5);
        ret = tr_read_tran( fp, tr );
    }

    fclose( fp );

    return( ret );
}


static int tr_read_tran( FILE *fp, TRANSACTION **tr )
{
    int ret,
        t;
    char    *p,
        *bkpt,
        buf[ BUFSIZE ];

    if( fgets( buf, BUFSIZE, fp ) == NULL )
    {
        return( ERR_READING );   ***//Failing here because of file not completeley arrived***
    }
   ....
   ...
   ret = //gets value for ret

    return( ret )
}

Upvotes: 0

Views: 167

Answers (4)

Viswesn
Viswesn

Reputation: 4880

The simple solution to this problem is

1) Changes in Producer code: "Make your producer (the creator of the file") to write the file contents to /tmp folder; and move the file from /tmp folder to the source folder;

                             [or]

2)

a) Change in Producer code: "Make your producer to write in source foleder with _part extension like "Myfile.txt._part" on completion of write operation then remove the last extension added to it; so the file name will be "Myfile.txt";

b) Changes in consumer code: "Make your consumer not to open any files that has last extension with ._part;

Upvotes: 0

Vladislav Varslavans
Vladislav Varslavans

Reputation: 2944

You could just create separate folder for files that are downloaded. Then just rename/move them to working directory.

Another way could be to append name with some "special" characters, that will tell service that file is still downloaded.

Upvotes: 1

Henrik
Henrik

Reputation: 2229

So many options..

here is a few:

  • use the filesystem events:

on linux: http://linux.die.net/man/7/inotify

on windows: https://msdn.microsoft.com/en-us/library/342da5th(v=vs.90).aspx

  • use a checksum on clients, and on server, to ensure file integrity, before processing.

  • create an error handler in your c program, that will ensure a retry on errors. think try/catch, except this doesnt actually exist in c, so instead use setjmp, and longjmp http://www.di.unipi.it/~nids/docs/longjump_try_trow_catch.html

Upvotes: 0

Neil
Neil

Reputation: 1066

You should set ret to something in tr_read_tran() function...

#define NOT_ERROR 100000   // any value not zero

static int tr_read_tran( FILE *fp, TRANSACTION **tr )
{
    int ret = NOT_ERROR,
        t;
    char    *p,
        *bkpt,
        buf[ BUFSIZE ];

    if( fgets( buf, BUFSIZE, fp ) == NULL )
    {
        return( ERR_READING );   ***//Failing here because of file not completeley arrived***
    }

    return( ret )
}

Currently it is not initialized to any value.

Upvotes: 0

Related Questions