Stories, essays, learning, and other considerations
By Jehan

ZFS: send entire storage pool incrementally, with resume

Published and last modified

If you have a pool you want to backup, or transfer to another machine, this is an important thing to know. It will transfer recursively all the filesystems and all the snapshots, in one command !

TL;DR

Let’s keep it short :

1
2
3
4
5
6
7

# 1. Make a snapshot of your whole pool on the source system
zfs snap -r mypool@mysnapshot

# 2. On the remote system, make sure you have a clean pool with no snapshots, etc

# 3. send !
zfs send -v -R mypool@mysnapshot | ssh user@remote-machine zfs receive -F -v -s -u -o mountpoint=none myremotepool

What does this do ?

  • -R will send the pool recursively, copying all properties
  • -F will allow destroy operations on the target pool
  • -v to see progress
  • -s save a resumable token
  • -u don’t mount the filesystems that are sent
  • -o mountpoint=none prevent mountpoint property to be sent

Note: I’m not sure what can pre-exist on the target pool without interfering. I had trouble because I had already manually transfered one of the filesystems and I had a snapshot for both that filesystem and for the pool. To make sure things would work, I destroyed them all. I suspect the pool’s snapshot might be the only one that needed to be removed.

If the transfer fails

Time to use that resume token !

Use this command on the target machine to retrieve the token:

1

zfs get all tank/dataset

Then re-run the command using the token:

1

zfs send -v -t <token> | ssh root@remote-machine receive -F -v -s myremotepool

What if I want to send new snapshots (increments) later on ?

The option you’ll need is -I, which, as the doc says :

1
2
3
4
5

-I snapshot
    Generate a stream package that sends all intermediary snapshots
    from the first snapshot to the second snapshot.  For example, -I @a fs@d
    is similar to -i @a fs@b; -i @b fs@c; -i @c fs@d.  The incremental source
    may be specified as with the -i option.

This is how it can be used:

1
2
3

zfs snap mypool@mysecondsnapshot
zfs send -v -R -I zstorage@mysnapshot zstorage@mysecondsnapshot \
    | ssh root@remote-machine "zfs recv -s -v -F zstorage";

It will send all snapshots of all filesystems, between your first and second pool snapshots. You just need to make sure no filesystem was changed on the destination since the first snapshot. Otherwise, you must rollback those changes using this command first (be careful, this command destroys any change that has happened since!):

1

zfs list -r  -o name | xargs -i zfs rollback -r {}@mysnapshot

References