Over the weekend I took the time to upgrade my file server from FreeNAS 9 to FreeNAS 11 (version 10 was an unreleased clusterf*ck). The big challenge wasn’t upgrading FreeNAS in place for the first time (I’ve done past upgrades by installing FreeNAS on new 16GB SanDisk Cruzer Fit flash drives). That part went quite smoothly. The big challenge was changing something that could jeopardize my data. By replacing the six-drive RAIDZ3 vdev with three two-drive stripped mirror vdevs, I had to destroy my data, restore my data from backup, and hope that my data wasn’t corrupted in the process.
The advantage of having a six-drive RAIDZ3 vdev is being able to lose three hard drives while still accessing the data (albeit with degraded performance until replacing the drives). While I never had multiple hard drives fail at the same time, I’ve had multiple hard drives fail one at a time from the oldest to the youngest. No sooner did the new drive finished rebuilding, the next drive starts signaling its imminent failure. The disadvantages of this configuration are not being able to recognize additional space until all six drives get replace with higher capacity drives, not being able to add another six-drive vdev, not getting the best read performance from a single vdev, and rebuilding a replacement drive takes forever.
The advantages of having three two-drive stripped mirror vdevs is being able to recognize additional space after upgrading two hard drives in a vdev, adding additional two-drive vdevs to increase overall storage, increasing read performance from multiple vdevs, and rebuilding a replacement drive at a faster rate. Since I’m using six out of eight hard drive slots inside my case, I need a SATA-III controller card and two 1TB Western Digital Red NAS 3.5″ drives to add another stripped mirror vdev. The disadvantage of this configuration is losing all data if two drives in a single vdev fail at the same time, which is more risky than losing more than three drives at the same time.
For my purposes, expansion and performance outweighs the risk of failure.
Before I could break down the old vdev to build the new vdevs, I had to ensure that my ~700GB of data got backed up to the Red Hat Linux box and the gaming rig. Backing up to the Red Hat Linux box took only a few minutes via rsync as this was the primary backup for the file server. Backing up to the gaming rig took longer as everything got copied over for the first time, taking over two hours at a 98MB per second transfer rate.
After the upgrade of FreeNAS got finished, I used the GUI to break down the existing configuration. Unfortunately, the GUI didn’t allow me to create stripped mirror vdevs. I had to do this from the command line: the create command created a two-drive stripped mirror vdev from hard drives /dev/ada0 and /dev/ada1, the two add commands created and added two more two-drive stripped mirror vdev from hard drives /dev/ada2 to /dev/ada5, and the status command displays the new configuration.
[creimer@oberon ~]$ zpool create storage mirror /dev/ada0 /dev/ada1
[creimer@oberon ~]$ zpool add storage mirror /dev/ada2 /dev/ada3
[creimer@oberon ~]$ zpool add storage mirror /dev/ada4 /dev/ada5
[creimer@oberon ~]$ zpool status storage
pool: storage
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
ada3 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
ada4 ONLINE 0 0 0
ada5 ONLINE 0 0 0
Restoring the data from the Red Hat Linux box took over seven hours at a 35MB per second transfer rate. The slow transfer speed comes from rsync calculating the checksums on a $25 AMD AM1 quad-core processor that is less powerful than the $50 AMD AM3 dual-core processor on the file server or the $100 AMD AM3 eight-core processor on the gaming rig. While it would have been faster to restore the data from the gaming rig, the secondary backup without the checksums may have been less reliable than the primary backup.
What changed with the new configuration? Not much. I still have 6TB of raw space and 3TB of usable space as before. The only noticeable change is copying large video files between the file server and gaming rig at a 108MB per second transfer rate (a 10% increase). I’m sure FreeNAS 11 has some significant features over FreeNAS 9, but I haven’t noticed those yet.