CODEWITHSUNDEEP
linuxstorageddparallel-io
Mike
Mike

Reputation: 1780

Fast Disk Cloning

Is there a way to have Linux read ahead when cloning a disk? I use the program named "dd" to clone disks. The last time I did this it seemed as though the OS was reading then writing but never at the same time. Ideally, the destination disk would be constantly writing without waiting that's of course if the source disk can keep up.

UPDATE: I normally choose a large block size when cloning (ex. 16M or 32MB).

Upvotes: 7

Views: 21964

Answers (7)

Paolinux
Paolinux

Reputation: 177

if the two disks use different channel (e.g., SATA) you can use high performance tool like fastDD. The authors claim:

"In this work, we reviewed the problem of reliably and efficiently copying data, recalling all the hardware and software mechanisms which intervene and interfer in the copying process. Our consideration have been coded in fastdd, a C++ program able to copy data very efficiently, as we show in our test."

Moreover the tool keeps a syntax very similar to the old dd.

http://www.dei.unipd.it/~zagonico/fastdd/

https://github.com/zagonico86/fastdd

Upvotes: 2

Thomas Kammeyer
Thomas Kammeyer

Reputation: 4507

Maybe you can use two processes

dd if=indevfile | dd of=outdevfile

I'll assume you can set the other dd options as it suits you. This has some overhead but should allow asynchrony between reading one disk and writing the other.

Upvotes: 1

SteveMenard
SteveMenard

Reputation: 69

The fastest for me:

dd if=/dev/sda bs=1M iflag=direct | dd of=/dev/sdb bs=1M oflag=direct

reaches ~100MiB/s, whereas other options (single process, no direct, default 512b block size, ...) don't even reach 30MiB/s...

To watch the progress, run in another console:

watch -n 60 killall -USR1 dd

Upvotes: 6

John Vasileff
John Vasileff

Reputation: 5478

Commodore Jaeger is right about:

dd if=/dev/sda of=/dev/sdb bs=1M

Also, adjusting "readahead" on the drives usually improves performance. The default may be something like 256, and optimal 1024. Each setup is different, so you would have to run benchmarks to find the best value.

# blockdev --getra /dev/sda
256
# blockdev --setra 1024 /dev/sda
# blockdev --getra /dev/sda
1024
# blockdev --help
Usage:
  blockdev -V
  blockdev --report [devices]
  blockdev [-v|-q] commands devices
Available commands:
    --getsz (get size in 512-byte sectors)
    --setro (set read-only)
    --setrw (set read-write)
    --getro (get read-only)
    --getss (get sectorsize)
    --getbsz    (get blocksize)
    --setbsz BLOCKSIZE  (set blocksize)
    --getsize   (get 32-bit sector count)
    --getsize64 (get size in bytes)
    --setra READAHEAD   (set readahead)
    --getra (get readahead)
    --flushbufs (flush buffers)
    --rereadpt  (reread partition table)
    --rmpart PARTNO (disable partition)
    --rmparts   (disable all partitions)
#

Upvotes: 10

Leon Timmermans
Leon Timmermans

Reputation: 30225

About your update: How big are the caches of your HDs? (specially the writing one). It may be that that is too much and you may need to reduce it to prevent unnecessary blocking.

Upvotes: 0

Commodore Jaeger
Commodore Jaeger

Reputation: 33350

You might try increasing the block size using the bs argument; by default, I believe dd uses a block size equal to the disk's preferred block size, which will mean many more reads and writes to copy an entire disk. Linux's dd supports human-readable suffixes:

dd if=/dev/sda of=/dev/sdb bs=1M

Upvotes: 7

Leon Timmermans
Leon Timmermans

Reputation: 30225

Are you sure it isn't doing that at the same time? I would expect the disk caches to make sure it that happens. If not, non-blocking or even asynchronous reads/writes may help,

Upvotes: 0

Related Questions