CODEWITHSUNDEEP
ruby-on-railsrubylinuxfile
Max Williams
Max Williams

Reputation: 32933

Store a checksum or similar for a file, to tell easily if it's the same as another file

In our app we have a table called support_files which stores documents that have been uploaded , which are mostly PDFs.

I'd like to get a unique list of these files, often the same file is uploaded more than once. I thought that a way to do this would be to add a column to the database called "checksum", and then, for each file, calculate the checksum somehow and store it in the column. (This is obviously the slow part).

Once this is done then I can easily filter out duplicates from my table by examining the checksum column.

Can anyone recommend a method to generate this checksum/hash/whatever? Ideally I'd like to generate a hash/checksum that's large enough to guarantee uniqueness, but small enough to fit into a string field in my database.

My server's running on Ubuntu server, and the total number of files I need to checksum is currently around 12,000. For the sake of argument assume it won't grow over 100,000.

A bit of Googling reveals sha1sum, but this may be more suited to telling if a file has been accidentally changed rather than if two files are different?

Upvotes: 1

Views: 514

Answers (1)

Brad Werth
Brad Werth

Reputation: 17647

Take a look at Digest::SHA256, it can interface directly with files and it works great.

From the referenced documentation:

p Digest::SHA256.file("X11R6.8.2-src.tar.bz2").hexdigest
# => "f02e3c85572dc9ad7cb77c2a638e3be24cc1b5bea9fdbb0b0299c9668475c534"

``

Upvotes: 2

Related Questions