What I want to do?
I use my pool to securely store backups, archive my old documents and keep huge family’s photo library.
I have new disks. They were tortured with
badblocks, so they’re ready to create ZFS pool.
I’ve read few documents about different approaches 1 2 3. I wanted to be sure if anything changed during past years. One of articles recommends mirroring over RAIDZ. Resilvering is faster, at the same time putting IO less stress on whole pool. But pool as small as mine, relies on single drive which might die in between and data won’t be recoverable. Eventually, I decided to go for RAIDZ1 for now and in the future I rather move to RAIDZ2. For that, I have to buy one more disk - Black Friday is close, we will see.
One thing that is available now, but was not, when I created ZFS pool last time, is encryption. I want it. I won’t be running with my PC anywhere, but in case if disk will be damaged and I will have to return it. No worries, data is unavailable. So I want it 😄
Let’s do it
I use Ubuntu 21.10 but despite installation of ZFS utils, rest of commands is not Ubuntu specific and should work on any distro.
sudo apt install -y zfsutils-linux zfs-zed
First, I have to generate secure key/file, that will be used to encrypt/decrypt file system. I will keep it on my root filesystem, actually in
/root directory, as it won’t be easily available for other users.
It’s possible to use password or raw file. On laptop, I will choose password, but on my desktop PC I prefer file. First, I was thinking to place it in
/etc/zfs but it’s world readable. So I decided I will drop it in
sudo dd if=/dev/random of=/root/.zfs-encrypt.key bs=1 count=32
I strongly advice to make backup of this key. Multiple backups actually :)
Create a fully encrypted pool
Now I can create pool. Most examples refer to disks as
sdb, etc. But I prefer to use their labels as drive model plus serial number. Just check
/dev/disk/by-id/. Other way is by use of
wwn-* ids, they’re also stable across restarts, cable changes, etc. I strongly recommend such ids over standard letters.
sudo zpool create \ -o ashift=12 \ -o feature@encryption=enabled \ -O encryption=on \ -O keylocation=file:///root/.zfs-encrypt.key \ -O keyformat=raw \ storage raidz1 \ ata-WDC_WD140EDGZ-11B1PA0_9MGJK4YK \ ata-WDC_WD140EDGZ-11B1PA0_Y6GVH40C \ ata-WDC_WD140EDGZ-11B1PA0_Y6GWHD3C
What happen here?
-o ashift=12- treat disks sector size as 4KB, it was detected automatically but I wanted to be sure :)
-o feature@encryption=enabled -O encryption=on -O keylocation=file:///root/.zfs-encrypt.key -O keyformat=raw- enable encryption on whole pool, use raw key and default encryption method:
Automatically decrypt on boot
So my pool is encrypted now, but I don’t want to decrypt it manually each time my PC starts. So I added such service:
sudo cat <<EOF > /etc/systemd/system/zfs-load-key.service [Unit] Description=Load encryption keys DefaultDependencies=no After=zfs-import.target Before=zfs-mount.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/sbin/zfs load-key -a StandardInput=tty-force [Install] WantedBy=zfs-mount.service EOF
It have to enabled it now:
sudo systemctl daemon-reload sudo systemctl enable zfs-load-key
In ArchLinux wiki 4, you can find example for service, that can load individual key per pool, but I had trouble getting it to work. Service above loads all keys for all pools and it’s matching my case of just two pools ;)
Let’s configure pool
There are few options worth to setup just after pool creation:
# automatically expand pool when new disk is added sudo zpool set autoexpand=on storage # automatically replace failed disk with hot spare sudo zpool set autoreplace=on storage # enable LZ4 compression sudo zfs set compression=lz4 storage # disable access time - for better performance sudo zfs set atime=off storage
For better explanation of options, check best practices on Aaron’s Toponce blog 5.
Create datasets (actual mount points)
Now I’m ready to create few datasets, that will be actually used to store files.
# photos are well compressed, so I don't need to compress them again sudo zfs create storage/photos sudo zfs set compression=off storage/photos # on backups I often just copy files staring, so let use stronger compression ZSTF sudo zfs create storage/backup sudo zfs set compression=zstd storage/backup # I like to keep my downloads on pool too sudo zfs create -o mountpoint=/home/timor/Downloads storage/downloads
Setting TLER on boot
Having disks in RAID, it’s good to have TLER enabled to rely on RAID for error recovery instead of internal hard drive recover 6. My disks support it, but it have to be enabled
sudo smartctl -l scterc /dev/sdd sudo smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.0-21-generic] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org SCT Error Recovery Control: Read: Disabled Write: Disabled
It’s there but disabled.
sudo smartctl -l scterc,70,70 /dev/sdd sudo smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.0-21-generic] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org SCT Error Recovery Control set to: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds)
Example below will overwrite /etc/rc.local. If you already use this file, please edit it on your own!
Make it persistent between restarts:
sudo cat <<EOF > /etc/rc.local #!/bin/bash # I have such disks, let's iterate over them # /dev/disk/by-id/ata-WDC_WD140EDGZ-11B1PA0_9MGJK4YK # /dev/disk/by-id/ata-WDC_WD140EDGZ-11B1PA0_Y6GVH40C # /dev/disk/by-id/ata-WDC_WD140EDGZ-11B1PA0_Y6GWHD3C for i in 9MGJK4YK Y6GVH40C Y6GWHD3C; do echo smartctl -l scterc,70,70 /dev/disk/by-id/ata-WDC_WD140EDGZ-11B1PA0_\$i > /dev/null; done EOF sudo chmod +x /etc/rc.local sudo systemctl enable rc-local.service
That’s it. All done.