- HCL 95.6%
- Shell 3.4%
- Nix 0.4%
- Makefile 0.3%
- Jinja 0.2%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
All checks were successful
/ prod-home (push) Successful in 50s
Reviewed-on: #72 |
||
| .forgejo | ||
| ansible | ||
| assets/wkd | ||
| docker/terraform | ||
| modules | ||
| scripts | ||
| workspaces | ||
| .envrc | ||
| .gitignore | ||
| devshell.toml | ||
| flake.lock | ||
| flake.nix | ||
| Makefile | ||
| README.md | ||
| renovate.json | ||
| shell.nix | ||
Homelab
My Terraform configurations for managing VMs in Proxmox, container workloads in Nomad, and all of the supporting services found below.
Overview
In order to achieve a resilient and immutable home lab, I use a number of Infrastructure as Code tools so everything is intentional and repeatable.
One of the self-imposed challenges that I set forth for my homelab was that I didn't want any machine to risk staying around so long as to invite major system drift that fell outside of this repository. For this reason the VMs that are created expire after 30 days and must be recreated from scratch. This forces me to place the changes that I care about within this repo instead of applying them piecemeal to each individual VM.
This homelab also has a unique quality in that if I need to perform any maintenance on any of the servers at home, workloads can be setup to float between servers at home and any cloud provider thanks to Nomad. This can also be utilized to add additional capacity if it is ever needed.
The Setup
This homelab uses Proxmox as the primary hypervisor at home and Vultr as a VPS provider within the cloud. When Terraform spins up a new VM, it creates a custom Cloud-init image for each VM which is seeded with the configuration it needs to:
- Install any dependant packages, certificates, and configurations.
- Connect to my Tailscale tailnet
- Configure Keepalived for LAN load balancing between home servers
- Automatically discover other Consul servers running on the tailnet to join
- Retrieve the rest of the provisioner secrets placed in Vault by Terraform, listed below.
- TLS certificates for internal services to be running on the nodes
- Vault tokens for Nomad
- Signed SSH host certificates
- Connect persistent storage from NAS (On home nodes only)
- Start Vault and Nomad in the overall cluster
Once Nomad has connected it automatically starts receiving jobs from the leader and system services like Traefik spin up with certificates stored in Vault. Traefik being up also makes the node eligible to start receiving traffic within Keepalived.
Core Components and Supporting Services
- Proxmox VE: KVM Hypervisor
- Terraform: Infrastructure as Code tool
- TrueNAS: Persistent network storage
- Tailscale: Flat mesh networking
- Consul: Service discovery and distributed KV store
- Vault: Secrets management service
- Nomad: Container and workload scheduler
- Let's Encrypt: Certificate Authority providing TLS for all for free
- Keepalived: Layer 4 load balancing and failover routing between servers
- Traefik: Application proxy
Repo Structure
.
├── docs # Documentation
│
├── ansible # Ansible scripts for post-provisioning
│
├── modules # Terraform modules
│ ├── cloudinit # Cloud-init provisioning scripts and helpers
│ ├── vultr # Vultr VM Provisioner (Uses cloudinit)
│ ├── proxmox_vm # Proxmox VM Provisioner (Uses cloudinit)
│ ├── nomad # Configure and run Nomad jobs with Terraform
│ │ └── jobspecs # Raw nomad job templates
│ └── vault # Vault backends, roles, and policies
│
└── workspaces # Terraform workspaces
├── prod-home # DNS, Nomad, and other home cluster glue
├── vultr # DNS, Nomad, and other Vultr cluster glue
├── lynx # Primary VM on Lynx hypervisor
└── lynx-aux # Secondary VM on Lynx hypervisor
└── lynx-quo # Tertiary VM on Lynx hypervisor
Hardware
| Name | Device | CPU | RAM | Primary Storage | Purpose |
|---|---|---|---|---|---|
| Lynx | 45Homelab HL-15 | AMD EPYC 7282 | 128GB DDR4 ECC | 4x10TB SATA HDD (RAIDZ2) | Proxmox Hypervisor |
| Nuc | Coffee Lake Intel NUC | 4-Core Intel i5-8259U | 32GB DDR4 | 1TB SATA SSD | Debian Forgejo Runner |
Roadmap
- Provision VMs via Cloud-init + Ansible
- Automated Terraform deployments via CI/CD
- Use a tool like Renovate to automate container and package upgrades