The lab

# how it fits together

One diagram for the whole shape of it. Traffic comes in from the top, through the edge and the firewall, across the 10GbE core, and down onto the compute and storage that run the workloads. The Pi fleet hangs off to the side on purpose, so home automation keeps working when the cluster is down for maintenance.

# the cluster

Two tiers on purpose. pve-00 and pve-01 are the heavy compute and GPU hosts, both on ECC memory, with discrete NVIDIA cards passed through to Kubernetes workers. The three HP small-form-factor boxes are consumer hardware doing honest work as quorum and Ceph OSD hosts. Not everything needs to be a server to be useful.

# three GPU generations

Three NVIDIA architectures, each passed straight through to a Talos Kubernetes worker. All three are time-sliced, so several workloads share a card instead of one pinning it. I looked at NVIDIA's MPS first and turned it down: it sets exclusive compute mode, which blocks the NVENC and NVDEC engines the transcoding and camera detection depend on. Transcode, local inference, and the occasional experiment all land here.

# storage

Everything sits on Ceph, on the order of 230 TiB raw spread across every node. Spinning disks make up the bulk pool, and NVMe backs the fast pool, the metadata, and local VM storage. The big NVMe counts hang off ASUS Hyper M.2 carrier cards using PCIe bifurcation. This is the layer that taught me the most, mostly the hard way, which is a story the blog gets into. Backups run to a dedicated Proxmox Backup Server, because "I have backups" and "I have tested restores" are not the same sentence.

# the fabric

A Brocade ICX6610 sits at the core, with 10GbE SFP+ uplinks to every node and jumbo frames end to end. Each node runs a two-bridge, VLAN-aware layout over two 10GbE links: one trunk carries tenant and VM traffic, the other isolates the latency-sensitive storage and cluster replication on its own physical port. That split is what lets a firewall VM live-migrate between nodes without dropping a packet. Routing between the dozen or so segments is handled by a high-availability OPNsense pair with CARP failover.

The per-node 10GbE cards are a museum of whatever was good and cheap at the time: Intel X550, Mellanox ConnectX-3 Pro and ConnectX-5, and Chelsio T320. They all do the same job. Edge switching and wireless are UniFi, and an APC UPS with a network card keeps the rack alive long enough to shut down cleanly.

# the pi fleet

A handful of Raspberry Pis run the things that should keep working even when the cluster is down for maintenance, mainly home automation and out-of-band tooling. They are named after Culture ships, which felt appropriate for machines whose entire job is to be quietly competent.

They share a small bag of USB radios that move between nodes as needed: a Z-Wave controller, a multiprotocol Zigbee/Thread stick, and a Google Coral TPU for edge ML.

# what it runs

The hardware is only half the story. Here's what runs on it, grouped and not exhaustive. It drifts as I add and retire things.