What does `native overlay diff` mean in overlay2 storage driver?

Question

I experienced that a lower layer(diff) of an image which is associated with the running container is removed. (So, some files in the container are removed)

I think 'Native Overlay Diff' option from docker info is quite suspicious.

My docker info like below:

$ docker info
...
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: false
...

Do you guys know the exact meaning of 'Native Overlay Diff'?

Answer 1

Docker doesn't like it when the Metadata only copy up option is enabled on the Overlay filesystem used by Docker volumes. This is a Linux kernel optimisation meant to reduce file copying when only the file metadata changes.

You can read more about it here: https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt

When metadata only copy up feature is enabled, overlayfs will only copy
up metadata (as opposed to whole file), when a metadata specific operation
like chown/chmod is performed. Full file will be copied up later when
file is opened for WRITE operation.
In other words, this is delayed data copy up operation and data is copied
up when there is a need to actually modify data.

You can disable this by creating the following file:

/etc/modprobe.d/overlay.conf

# Used by docker to avoid issue:
#  Not using native diff for overlay2, this may cause degraded performance for building images: 
#  kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled

options overlay metacopy=off
options overlay redirect_dir=off

This passes the relevant parameters to the overlay module. Restart your system and check if the change applied:

> cat /sys/module/overlay/parameters/redirect_dir
N

And

> docker info|grep -i 'overlay diff'
Native Overlay Diff: true

Answer 2

tmp fix

echo 0 > /sys/module/overlay/parameters/redirect_dir

then restart docker

systemctl restart docker 

show docker info

docker info
...
  Native Overlay Diff: true
...

use this cmd to enable Native Overlay Diff

Answer 3

This seems to be related to the OVERLAY_FS_REDIRECT_DIR kernel option, which is described in Kconfig as:

config OVERLAY_FS_REDIRECT_DIR
bool "Overlayfs: turn on redirect dir feature by default"
depends on OVERLAY_FS
help

If this config option is enabled then overlay filesystems will use redirects when renaming directories by default. In this case it is still possible to turn off redirects globally with the "redirect_dir=off" module option or on a filesystem instance basis with the "redirect_dir=off" mount option.

Note, that redirects are not backward compatible. That is, mounting an overlay which has redirects on a kernel that doesn't support this feature will have unexpected results.

If unsure, say N.

Some discussion on moby issues 34342 and 34320 indicates that, if all of the following are true:

1. the OVERLAY_FS_REDIRECT_DIR kernel option is enabled
2. the storage engine is overlay2 (the default in most cases)
3. docker is storing images on a file system that is not mounted with redirect_dir=off
4. the "native" diff driver is used

5. a nonempty directory is renamed as part of a docker build, e.g. in a Dockerfile like the following:

   FROM busybox
   
   RUN mkdir /dir1
   RUN touch /dir1/newfile
   RUN mv /dir1 /dir2
   

Then the resulting image will not properly record the contents of the renamed directory (i.e., dir2 will not contain newfile) because the directory rename was implemented as a redirect using an extended file attribute (xattr) which is not understood by the docker archiving process. To solve this problem, when the first three conditions above are met, then docker will use the "naive" diff driver which produces correct images, but is slower than the "native" diff driver.

It seems like it is safe to ignore the warning, but if you notice slow builds, then you could try remounting the volume serving /var/lib/docker with the option redirect_dir=off.