CODEWITHSUNDEEP
hadoopmapreduce
Julias
Julias

Reputation: 5892

how to prevent hadoop corrupted .gz file

I'm using following simple code to upload files to hdfs.

FileSystem hdfs = FileSystem.get(config);
hdfs.copyFromLocalFile(src, dst);

The files are generated by webserver java component and rotated and closed by logback in .gz format. I've noticed that sometimes the .gz file is corrupted.

 > gunzip logfile.log_2013_02_20_07.close.gz 
 gzip: logfile.log_2013_02_20_07.close.gz: unexpected end of file

But the following command does show me the content of the file

> hadoop fs -text /input/2013/02/20/logfile.log_2013_02_20_07.close.gz

The impact of having such files is quite disaster - since the aggregation for the whole day fails, and also several slave nodes is marked as blacklisted in such case.

What can I do in such case? Can hadoop copyFromLocalFile() utility corrupt the file? Does anyone met similar problem ?

Upvotes: 1

Views: 1660

Answers (1)

Chris White
Chris White

Reputation: 30089

It shouldn't do - this error is normally associated with GZip files which haven't been closed out when originally written to local disk, or are being copied to HDFS before they have finished being written to.

You should be able to check by running an md5sum on the original file and that in HDFS - if they match then the original file is corrupt:

hadoop fs -cat /input/2013/02/20/logfile.log_2013_02_20_07.close.gz | md5sum
md5sum /path/to/local/logfile.log_2013_02_20_07.close.gz

If they don't match they check the timestamps on the two files - the one in HDFS should be modified after the local file system one.

Upvotes: 1

Related Questions