CODEWITHSUNDEEP
backupzfs
Xdg
Xdg

Reputation: 1835

Why received ZFS dataset uses less space than original?

I have a dataset on the server1 that I want to back up to the second server2.

Server1 (original):

zfs list -o name,used,avail,refer,creation,usedds,usedsnap,origin,compression,compressratio,refcompressratio,mounted,atime,lused storage/iscsi/webhost-old produces:

NAME                        USED  AVAIL  REFER  CREATION                USEDDS  USEDSNAP  ORIGIN  COMPRESS  RATIO  REFRATIO  MOUNTED  ATIME  LUSED
storage/iscsi/webhost-old  67,8G  1,87T  67,8G  Út kvě 31  6:54 2016   67,8G       16K  -            lz4  1.00x     1.00x        -      -  67,4G

Sending volume to the 2nd server:

zfs send storage/iscsi/webhost-old | pv | ssh -c arcfour,[email protected] [email protected] zfs receive -Fduv pool/bkp-storage

received 69,6GB stream in 378 seconds (189MB/sec)

Server2 zfs list produces:

NAME                                 USED  AVAIL  REFER  CREATION               USEDDS  USEDSNAP  ORIGIN  COMPRESS  RATIO  REFRATIO  MOUNTED  ATIME  LUSED
pool/bkp-storage/iscsi/webhost-old  36,1G  3,01T  36,1G  Pá pro 29 10:25 2017   36,1G         0  -            lz4  1.15x     1.15x        -      -  28,4G

Why is there such a difference in sizes? Thanks.

Upvotes: 3

Views: 2025

Answers (1)

Dan
Dan

Reputation: 7737

From what you posted, I noticed 3 things that seemed odd:

  1. the compressratio is 1.15x on system 2, but 1.00x on system 1
  2. on system 2, used is 1.27x higher than logicalused
  3. the logicalused and the number zfs receive report are ~2.3x higher on system 1 than system 2

These terms are all defined in the man page, but are still confusing to reverse-engineer explanations for in practice.

(1) could happen if you enabled compression on the source dataset after you wrote all the data to it, since ZFS doesn't rewrite the data to compress it when you enable that setting. The data sent by zfs send is uncompressed unless you use -c, but system 2 will try to compress it as it runs zfs receive if the setting is enabled on the destination dataset. If both system 1 and system 2 had the same compression settings before the data was written, they would have the same compressratio as well.

(2) can happen due to metadata written along with your data, but in this case it's too high for "normal" metadata, which accounts for 1-2% of most pools. It's probably caused by a pool-wide setting, like configuring RAID-Z, or a weird combination of striping and mirroring (like 4 stripes, but with one of them being a mirror).

For (3), I re-read the man page to try to figure it out:

 logicalused

 The amount of space that is "logically" consumed by this dataset and
 all its descendents.  See the used property.  The logical space
 ignores the effect of the compression and copies properties, giving a
 quantity closer to the amount of data that applications see.

If you were sending a dataset (instead of a single iSCSI volume) and the send size matched system 2's logicalused value (instead of system 1's), I would guess you forgot to send some child datasets (i.e. by using zfs send -R). However, neither of those are true in this case.

I had to do some additional digging -- this blog post from 2005 might contain the explanation. If system 1 didn't have compression enabled when the data was written (like I guessed above for (1)), the function responsible for not writing zeroed-out blocks (zio_compress_data) would not be run, so you probably have a bunch of empty blocks written to disk, and accounted for in the logicalused size. However, since lz4 is configured on system 2, it would run there, and those blocks would not be counted.

Upvotes: 3

Related Questions