My homelab in a Saigon apartment: 3 node Proxmox, TrueNAS, 41 container, 186W

Many people ask in my Self Hosted topic "what do you actually run", so here is the full tour. It start 4 years ago as one old laptop under the TV. Now it is a 12U wall cabinet in the corner of our spare room, and my wife call it "the heater".

Hardware

• pve1, pve2: Lenovo M720q Tiny, i5-8500T, 32GB RAM, 1TB NVMe each. Bought second hand, very good value
• pve3: HP EliteDesk 800 G4 Mini, i5-8500T, 32GB, 512GB NVMe
• NAS: TrueNAS SCALE bare metal, i3-9100 on a C246 board, 32GB ECC, 4 × 8TB in two ZFS mirrors (more about this in another post)
• Network: MikroTik hEX S as router, TP-Link 8 port managed switch, VLANs for LAN / IoT / cameras / management
• UPS: APC Back-UPS 1000VA
• Cabinet: 12U wall cabinet with 2 fans on top, silica gel boxes inside (I will explain)

What runs (41 containers + 4 VM)

• Media: Jellyfin, Immich (all family photos, 1.1TB)
• Docs and passwords: Vaultwarden, Paperless-ngx
• Home: Home Assistant (VM), Frigate for 3 cameras, Mosquitto
• Dev: Gitea, Woodpecker CI, a Postgres for my side projects
• Infra: Traefik, Authentik for SSO, Pi-hole #2 (backup of the one on the Pi), Uptime Kuma, Prometheus + Grafana + node exporter on everything
• Nextcloud. I regret this one, see below

Proxmox cluster notes

• 3 nodes because of quorum. With 2 nodes, when one die the other one lose quorum and will not start VMs. You can use a QDevice on a Pi instead of a third node, it is fine too
• Corosync on its own VLAN. Do not put cluster traffic on the same link as a big backup job
• No Ceph. Ceph on 1GbE and consumer SSD is pain, I try for 3 weeks, give up. I use ZFS on each node and Proxmox storage replication every 15 minutes for the important VMs (Vaultwarden, Home Assistant). If a node die, I lose max 15 minutes and HA start the VM on another node

Techno Tim's tour make me feel very poor, but it give me the idea for the VLAN layout.

Power and money

I measure with a smart plug at the wall for 3 months: 186W average. That is around 134 kWh per month.

The painful part is the EVN tiered price. Our home already use ~350 kWh a month without the lab (aircon, it's Saigon), so every lab kWh is billed in the top tiers, around 3,000-3,400 VND per kWh. So the lab cost me about 420,000-450,000 VND a month (about 17 USD). My wife know this number exactly, she bring it up when I want to buy a new disk.

Humidity, the real enemy

HCMC is 75-85% humidity most of the year. In year one I lost one HDD and one cheap switch, and when I open the switch there was corrosion on the board. Now:

• The room has a small dehumidifier set to 55%
• Silica gel boxes inside the cabinet, I dry them in the oven every month (wife love this also)
• Cabinet fans always on, the cabinet is never closed air

What I would not buy again

• Cheap QLC SSD for Proxmox boot + VM storage. ZFS + Proxmox logging write a lot. One drive go from 0% to 23% "Percentage Used" in 8 months. Check yours: smartctl -a /dev/nvme0 | grep -i "percentage used"
• Nextcloud. Too heavy, too many moving parts for what my family use. Immich for photos + Syncthing for files is lighter and my wife like it more
• A 10GbE switch "for the future". It is the future now and I still use 1GbE for everything. It sit in the box

Next post I will write about the ZFS mirror and the night a drive die. hope this help :)