Skip to content
article

ZFS mirror NAS: what I learn when a drive die at 2am (humidity edition)

Commands, the alert that went nowhere, and why I keep a cold spare on the shelf

Nguyễn Minh Hiếu@homelab_hieu

July 18, 2026 · 3 min read

#1 in Self Hosted

Contents(5)

Last month one of my 8TB drives die at 2am on a Tuesday. Nothing was lost, because ZFS mirror, but I learn a few things that I want to write down before I forget.

The pool

TrueNAS SCALE, one pool called tank, 4 × 8TB in two mirror vdevs. Each mirror is one WD Red Plus + one Seagate IronWolf, bought from different shops in different months, so they are not from the same batch.

TrueNAS does this in the UI, but this is what happen underneath:

zpool create -o ashift=12 tank \
  mirror /dev/disk/by-id/ata-WDC_WD80EFZZ-xxxx /dev/disk/by-id/ata-ST8000VN004-xxxx \
  mirror /dev/disk/by-id/ata-WDC_WD80EFZZ-yyyy /dev/disk/by-id/ata-ST8000VN004-yyyy

zfs set compression=lz4 tank
zfs set atime=off tank

Always use /dev/disk/by-id/, never /dev/sda. sda can become sdb after reboot.

Why mirrors and not RAIDZ

  • Resilver is much faster (only copy from one disk, not read every disk in the vdev)
  • I can grow the pool by adding 2 disks, not 4 or 5
  • Better IOPS for VMs and Immich thumbnails
  • Cost: 50% space. For home, I accept

The night

I find out at 7am, not 2am, because Uptime Kuma show the NAS "slow". The TrueNAS email alert went to an old Gmail that I don't read anymore. Lesson 1.

$ zpool status -x
  pool: tank
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
...
	  mirror-1                   DEGRADED     0     0     0
	    ata-WDC_WD80EFZZ-yyyy    ONLINE       0     0     0
	    ata-ST8000VN004-yyyy     FAULTED     41     0     0  too many errors

SMART on the bad one:

smartctl -a /dev/sdc | grep -E "Reallocated|Pending|Uncorrectable"
  5 Reallocated_Sector_Ct   ...   312
197 Current_Pending_Sector  ...   88
198 Offline_Uncorrectable   ...   88

Dead. When I take it out, the PCB connector side have a little bit of white corrosion. Humidity again.

Replace

I have a cold spare on the shelf (lesson 2, see below), so:

zpool offline tank ata-ST8000VN004-yyyy
# power off, swap disk, power on
zpool replace tank ata-ST8000VN004-yyyy /dev/disk/by-id/ata-ST8000VN004-zzzz
zpool status tank   # watch the resilver

Resilver 5.1TB used took about 9 hours. During resilver the other disk in that mirror is the only copy, so I stop all big jobs and pray a little bit.

Lessons

  1. Test your alerts. Send a test email every month. If the alert goes nowhere, you don't have alert
  2. Keep a cold spare. At 2am in Saigon you cannot buy an 8TB NAS drive, and the shop I trust in District 10 was out of stock for a week after. Without spare, I am one disk away from losing that vdev for a whole week
  3. Scrub monthly. TrueNAS default is every 35 days, keep it. Scrub find silent errors while the other copy is still good
  4. SMART long test scheduled every 2 weeks, short test daily
  5. Mix batches in each mirror. Same batch drives like to die together
  6. Humidity: dehumidifier in the room at 55%. Seriously
  7. RAID is not backup. Mirror protect you from a dead disk, not from rm -rf, ransomware, or a fire. My offsite is restic to Backblaze B2 (another post)

hope this help :)

#nas#homelab#linux#self-hosting

Written by

Nguyễn Minh Hiếu

Proxmox, Docker, ZFS, and a power bill my wife asks about. HCMC 🇻🇳 Backend dev by day, homelab by night. Founder of Self Hosted. Back up your data, seriously.

View profile

Get the best of #self_hosted weekly

One email each Monday with the most useful posts. No spam; unsubscribe in one click.

Comments

7

Sign in to join the conversation.

Sign in to comment
Alasdair Reid@onprem_or_die#3 in Self Hosted
good post. mirrors over raidz for home every time, the resilver alone is worth it. two things to add. one, the alert email: set up a second path. i have truenas alerts going to email AND a script that hits a healthchecks style url, if either stops i hear about it. two, when you buy the spare, burn it in before it goes on the shelf. full badblocks pass or a long smart test. a spare that's DOA is worse than no spare because you trusted it
3 net votes
Nguyễn Minh Hiếu@homelab_hieu#1 in Self Hosted
burn in the spare, very good point. I only plug it and check SMART short. will do badblocks on the next one
2 net votes
Graham Whitfield@greybeard_graham#2 in Self Hosted
Mixing batches is the advice nobody listens to until the day two drives with consecutive serial numbers die in the same week. I have seen it three times in my career. Good post. Cheers, G.
0 net votes
Chaminda Perera@chaminda_sysops#4 in Self Hosted
The email alert going to an old mailbox, I am telling you, this happens in banks also. We found once that the SAN alerts went to a person who left in 2016. Good honest post.
6 net votes
Prashanth Gowda@prashanth_sre#8 in Self Hosted
9 hours of resilver with one copy left is the scariest 9 hours in homelab. saving this
4 net votes
Jordan Pike@jpike2 rep in Self Hosted
"If the alert goes nowhere, you don't have alert" should be on a t-shirt. Nice writeup.
1 net votes
Lena Hoffmann@arch_btw_lena2 rep in Self Hosted
ashift=12 always, even if the disk says 512 byte sectors. learned this the hard way with a pool i could not fix without rebuilding
3 net votes