CODEWITHSUNDEEP
javajarzip
Mike Miller
Mike Miller

Reputation: 2318

Binary Difference in Zip/Jar file

It seems like building a jar or zip from the exact same source files will always yield a different file. I tried this both using the java jar command, and the jar and zip tasks in ant.

It appears to be due to the fact that new jars/zips have the timestamp set to the current time on each stored file.

Is there a way to force a zip tool to simply use the timestamp on the file on the filesystem to ensure that a jar built from the exact same source will appear exactly the same?

Upvotes: 10

Views: 18278

Answers (8)

Ajax
Ajax

Reputation: 2520

fwiw, anyone using the gradle build system can get a binary-stable jar cheaply / easily:

tasks.withType(Jar).configureEach {
    Jar jar ->
        jar.preserveFileTimestamps = false
        jar.reproducibleFileOrder = true
}

This does what the other answers suggest: make sure the timestamps are stable, AND make sure the order of the entries in the jar are also stable. I believe the brains here are outside of the jar command itself, but I'm not invested enough in this problem to check.

Upvotes: 1

Hartmut Schorrig
Hartmut Schorrig

Reputation: 549

This answer is insufficent. Read my other answer. I have not removed this answer because it shows some reason for non binary-compatibility, but not all reasons.

I have an elaborately answer, but unfortunately in german: https://www.vishia.org/SwEng/pdf/GenerateRepeatability_de.pdf Short presentation: 

echo compile javac
$JAVAC_HOME/bin/javac -d $TMPJAVAC/binjar -cp $CLASSPATH -sourcepath $SRCPATH $FILE1SRC 
mkdir $TMPJAVAC/binjar/META-INF
##Note: create the manifest file manually, not with jar, because of time stamp
cp $MANIFEST $TMPJAVAC/binjar/META-INF/MANIFEST.MF  
echo touch timestams to $VERSION
find $TMPJAVAC/binjar -exec touch -d $VERSION {} \;
echo build jar
$JAVAC_HOME/bin/jar -cvfM $JARFILE -C $TMPJAVAC/binjar . > $TMPJAVAC/jar.txt
if ! test "$MD5FILE" = ""; then echo output MD5 checksum
  md5sum -b $JARFILE > $MD5FILE
fi  
echo ok $JARFILE

It is a universal shell script controlled by script variable set from outside. It can be called from another shell script or from gradle. The essential is the touch command and the copy command for the manifest, and the M option (not m) for jar. Binary differences in jar file came from the time stamps of the contained files. Hartmut Schorrig

Upvotes: 0

Hartmut Schorrig
Hartmut Schorrig

Reputation: 549

Unfortuantelly my answer from 2020-03-17 does not produce a reproducible jar file (binary compatible) in all situations. The reason: The order of files in the jar depends on randomness. Generation on another maschine, I have used Windows and Linux, produces another file order in the jar/zip file which results in another binary content. If the jar files are compared file by file (after unzip), they are identically. But the pure binary jar is not so. I have a solution which does not use the jar command from the JDK but an own jar algorithm using the capability of JRE. The JRE contains java.util.jar.* and java.util.zip.* as standard. One can read the description in https://vishia.org/Java/html5/source+build/reproducibleJar.html. You can download a small https://www.vishia.org/Java/Download/versionArchive/vishiaZipJar-2020-03-23.jar with given MD5 check sum (visit this Download/versionArchive page). The article contains examples. I have tested with Windows and Linux, which different JDK versions, the results are binary compatible.

Upvotes: 0

András
András

Reputation: 21

Using the Java java.util.zip.ZipOutputStream standard library utility it is possible to create zip files with reproducible content.

The only trick is that the timestamp of the zip entries must be fixed using this trick:

ZipOutputStream zos=...;
ZipEntry ze=new ZipEntry("Filename");
zipEntry.setTime(0);
zos.putNextEntry(ze);
try
{
   zos.write(data);
}finally
{
  zos.closeEntry();
}

Upvotes: 2

Kees
Kees

Reputation: 41

The binary difference is because of the timestamp of the manifest files. If you let jar create a manifest itself it will create a manifest on the fly and set the created manifest to currentTimeMillis.

You can solve it by:
1. Do not add a manifest (if your using ant you must use zip instead of jar)
2. Add the manifest like you add normal files. (So the manifest is a file on your filesystem and it isn't created on the fly)

Upvotes: 4

Mike Miller
Mike Miller

Reputation: 2318

Ok, a coworker and I came up with a solution that works for us.

Instead of reengineering our entire build process to not delete any class or jar files, we use this procedure:

  1. Build new artifacts.
  2. Use jardiff (part of jnlp) to compare changes from previous build.
  3. If the diff jar that jardiff produces has no changes, get artifact from previous build.

Yeah, I know it sounds kludgy, but it sure beats rewriting build script to take this into account. Also, we can do a completely clean build on a fresh machine (in the case of server failure), and this process will ensure that only actually updated jars are produced.

Upvotes: 1

Gadi
Gadi

Reputation: 482

I had a similar problem and, as pjz suggests, I solved it by 'touch'-ing the files before adding them to the jar (so, it worked for me :-) ). You can find touch for Windows, if you need it, either in the GNU Windows Utilities, core utils: http://gnuwin32.sourceforge.net/packages/coreutils.htm , but it's a big package for just this single one (though there are many other useful utilities there you may like), or alternatively, download something like http://www.softpedia.com/progClean/Touch-for-Windows-Clean-41086.html .

Upvotes: 1

pjz
pjz

Reputation: 43057

I don't think there's a way to make zip do that, but you could certainly hammer the timestamps on the files on the filesystem to a known date (using the 'touch' command under unix - I dunno what under Windows) before you create the jar.

Upvotes: 1

Related Questions