tdpeuter/bos55-nix-config-cicd
1
Fork 0
forked from Bos55/nix-config
Code Pull requests 1 Releases Packages Activity Actions
bos55-nix-config-cicd/docs/binary-cache/implementation_plan.md

288 lines
14 KiB
Markdown
Raw Blame History

# NixOS CI/CD Automated Deployment with deploy-rs
## Overview
Implement a push-based automated deployment pipeline using **deploy-rs** for the homelab NixOS fleet. The pipeline builds on every push/PR, deploys on merge to `main`, and supports `test-<hostname>` branches for non-persistent trial deployments.
---
## Architecture
```
┌─────────────┐ push ┌──────────────────┐
│ Developer │────────────▶│ Forgejo (Git) │
└─────────────┘ └────────┬─────────┘
│
┌────────────────┼────────────────┐
▼ ▼ ▼
┌─────────────┐ ┌───────────┐ ┌──────────────┐
│ CI: Build │ │ CI: Check │ │ CI: Deploy │
│ all hosts │ │ flake + │ │ (main only) │
│ (every push)│ │ deployChk │ │ via deploy-rs│
└──────┬──────┘ └───────────┘ └──────┬───────┘
│ │ SSH
▼ ▼
┌─────────────┐ ┌──────────────────┐
│ Harmonia │◀─── push ────│ Target Hosts │
│ Binary Cache│─── pull ────▶│ (NixOS machines) │
└─────────────┘ └──────────────────┘
```
---
## Key Design Decisions
### Test branch activation (`test-<hostname>`)
deploy-rs's `activate.nixos` calls `switch-to-configuration switch` by default, which updates the bootloader. For test branches, we create a **separate profile** using `activate.custom` that calls `switch-to-configuration test` instead — this activates the configuration immediately but **does not update the bootloader**. On reboot, the host falls back to the last `switch`-deployed generation.
Magic rollback still works on test deployments: deploy-rs confirms the host is reachable after activation and auto-reverts if it can't connect.
```nix
# Test activation: active now, but reboot reverts to previous boot entry
activate.custom base.config.system.build.toplevel ''
cd /tmp
$PROFILE/bin/switch-to-configuration test
''
```
### Zero duplication in `flake.nix`
Use `builtins.mapAttrs` over `self.nixosConfigurations` to generate `deploy.nodes` automatically. Host metadata (IP, whether to enable deploy) is stored once per host config.
### Renovate bot compatibility
The pipeline is fully compatible with Renovate:
- **Minor/patch updates**: Renovate opens a PR → CI builds all hosts → Renovate auto-merges → CI deploys (uses `switch`, updates bootloader)
- **Major updates**: Renovate opens PR → CI builds → waits for manual review → merge → deploy with `switch` (persists across reboot)
- The deploy step differentiates using the **branch name**, not the commit source, so Renovate PRs behave identically to human PRs
### System version upgrades (kernel, etc.)
When a deployment requires a reboot (e.g., kernel upgrade):
1. CI deploys with `--boot` flag → calls `switch-to-configuration boot` (sets new generation as boot default without activating)
2. A separate reboot step (manual or scheduled) activates the change
> [!IMPORTANT]
> deploy-rs does not auto-detect whether a reboot is needed. The workflow can check if the kernel or initrd changed and conditionally use `--boot` instead, or always use `switch` and document that the operator should reboot when `nixos-rebuild` would have shown `reboot required`.
---
## Security & Trust Boundaries
### Trust model diagram
```
┌─────────────────────────────────────────────────────┐
│ TRUST ZONE 1 │
│ Developer Workstations │
│ • Holds sops-nix age keys (decrypt secrets) │
│ • Holds GPG/SSH keys for signed commits │
│ • Can manually deploy via deploy-rs │
│ • Can push to any branch │
└──────────────────────┬──────────────────────────────┘
│ git push (signed commits)
▼
┌─────────────────────────────────────────────────────┐
│ TRUST ZONE 2 │
│ Forgejo + CI Runner │
│ • Holds CI SSH deploy key (DEPLOY_SSH_KEY secret) │
│ • Does NOT hold sops-nix age keys │
│ • Branch protection: main requires PR + checks │
│ • Can only deploy via the deploy user │
│ • Builds are sandboxed in Nix │
└──────────────────────┬──────────────────────────────┘
│ SSH as "deploy" user
▼
┌─────────────────────────────────────────────────────┐
│ TRUST ZONE 3 │
│ Target NixOS Hosts │
│ • deploy user: system user, no shell login │
│ • sudo: ONLY nix-env --set and │
│ switch-to-configuration (NOPASSWD) │
│ • No write access to /etc, home dirs, etc. │
│ • sops secrets decrypted at activation via host │
│ age keys (not CI keys) │
└─────────────────────────────────────────────────────┘
```
### What each actor can do
| Actor | Can build | Can deploy | Can decrypt secrets | Can access hosts |
|---|---|---|---|---|
| Developer | ✅ | ✅ (manual) | ✅ (personal age keys) | ✅ (personal SSH) |
| CI runner | ✅ | ✅ (deploy user) | ❌ | Limited (deploy user) |
| deploy user | ❌ | ✅ (sudo restricted) | ❌ | N/A (runs on host) |
| Host age key | ❌ | ❌ | ✅ (own secrets only) | N/A |
### Hardening measures
1. **Branch protection** on `main`: require PR, passing checks, optional signed commits
2. **deploy user** ([`users/deploy/default.nix`](file:///c:/Users/tibod/Documents/projects/Bos55/bos55-nix-config-cicd/users/deploy/default.nix)): restricted sudoers, no home dir, system user
3. **CI secret isolation**: SSH key only, no age keys in CI — secrets are decrypted on-host at activation time by sops-nix using host-specific age keys
4. **Magic rollback**: if a deploy renders the host unreachable, deploy-rs auto-reverts within the confirm timeout
5. **`nix flake check` + `deployChecks`**: validate the flake structure and deploy configuration before any deployment
> [!NOTE]
> The deploy user SSH key is stored as a Forgejo Actions secret. Even if the CI runner is compromised, the attacker can only push Nix store paths and trigger `switch-to-configuration` — they cannot decrypt secrets, access user data, or escalate beyond what the restricted sudoers rules allow.
---
## Proposed Changes
### 1. Flake configuration
#### [MODIFY] [flake.nix](file:///c:/Users/tibod/Documents/projects/Bos55/bos55-nix-config-cicd/flake.nix)
- Add `deploy-rs` to flake inputs
- Auto-generate `deploy.nodes` from `self.nixosConfigurations` using `mapAttrs` — **zero duplication**
- Add `checks` output via `deploy-rs.lib.deployChecks`
- Define a helper that reads each host's `config.networking` for hostname/IP
```nix
# Sketch of the deploy output (no per-host duplication)
deploy.nodes = builtins.mapAttrs (name: nixos: {
hostname = nixos.config.homelab.deploy.targetHost; # defined per host
sshUser = "deploy";
user = "root";
magicRollback = true;
autoRollback = true;
profiles.system = {
path = deploy-rs.lib.x86_64-linux.activate.nixos nixos;
};
}) (lib.filterAttrs
(name: nixos: nixos.config.homelab.users.deploy.enable or false)
self.nixosConfigurations);
```
---
### 2. Deploy user module
#### [MODIFY] [default.nix](file:///c:/Users/tibod/Documents/projects/Bos55/bos55-nix-config-cicd/users/deploy/default.nix)
- Add option `homelab.deploy.targetHost` (string, the IP/hostname for deploy-rs to SSH into)
- Support multiple SSH authorized keys (CI key + personal workstation keys)
- Add `nix.settings.trusted-users` option for the deploy user (needed for `nix copy` from cache)
---
### 3. Enable deploy user on target hosts
#### [MODIFY] Host `default.nix` files (per host)
- Enable `homelab.users.deploy.enable = true` on all deployable hosts
- Set `homelab.deploy.targetHost` to each host's IP (e.g., `"192.168.0.10"` for Ingress)
- Currently only `Niko` has deploy enabled; extend to all non-`Template` hosts
---
### 4. Binary cache (Harmonia)
#### [NEW] [modules/services/harmonia/default.nix](file:///c:/Users/tibod/Documents/projects/Bos55/bos55-nix-config-cicd/modules/services/harmonia/default.nix)
- Create `homelab.services.harmonia` module wrapping `services.harmonia`
- Generates a signing key pair for the cache
- Configures Nginx reverse proxy with HTTPS (via ACME or internal cert)
- All hosts configured to use the cache as a substituter via `nix.settings.substituters`
> [!TIP]
> Harmonia is chosen over attic (simpler, no database needed) and nix-serve (better performance, streaming, zstd compression). It serves your `/nix/store` directly, so the CI runner can `nix copy` built closures to the cache host after a successful build.
#### [NEW] [modules/common/nix-cache.nix](file:///c:/Users/tibod/Documents/projects/Bos55/bos55-nix-config-cicd/modules/common/nix-cache.nix)
- Configure all hosts to use the binary cache as a substituter
- Add the cache's public signing key to `trusted-public-keys`
- Usable by personal devices too (add the cache URL + public key to their `nix.conf`)
---
### 5. CI Workflows
#### [MODIFY] [build.yml](file:///c:/Users/tibod/Documents/projects/Bos55/bos55-nix-config-cicd/.github/workflows/build.yml)
- Use the dynamic `determine-hosts` job output for the build matrix (already partially implemented)
- Add `nix flake check` step for deployChecks validation
- Build all hosts on every push/PR
- Optionally push built closures to the Harmonia cache after successful build
#### [NEW] [deploy.yml](file:///c:/Users/tibod/Documents/projects/Bos55/bos55-nix-config-cicd/.github/workflows/deploy.yml)
- Trigger: push to `main` or `test-*` branches (after build passes)
- Load `DEPLOY_SSH_KEY` from Forgejo Actions secrets
- **For `main`**: `deploy .` (all hosts, `switch-to-configuration switch`)
- **For `test-<hostname>`**: deploy only the matching host with a **test profile** (`switch-to-configuration test`) — no bootloader update
- Magic rollback enabled by default
- Optional `--boot` mode for kernel upgrades (triggered by label or manual dispatch)
#### [NEW] [check.yml](file:///c:/Users/tibod/Documents/projects/Bos55/bos55-nix-config-cicd/.github/workflows/check.yml)
- Runs `nix flake check` (includes deployChecks)
- Runs `nix eval` to validate all configurations parse correctly
- Can be required as a status check for Renovate auto-merge rules
---
### 6. Monitoring
#### [NEW] [modules/services/monitoring/default.nix](file:///c:/Users/tibod/Documents/projects/Bos55/bos55-nix-config-cicd/modules/services/monitoring/default.nix)
- Enable node exporter on all hosts for Prometheus scraping
- Export NixOS generation info: current generation, boot generation, system version
- Optionally integrate with the existing infrastructure (e.g., Prometheus on Production)
Script/service to export NixOS deploy state:
```bash
# Metrics like:
# nixos_current_generation{host="Niko"} 42
# nixos_boot_generation{host="Niko"} 42 # same = no pending reboot
# nixos_config_age_seconds{host="Niko"} 3600
```
When `current_generation != boot_generation`, the host has a test deployment active (or needs a reboot).
---
### 7. Local VM Testing
#### [NEW] [test/vm-test.nix](file:///c:/Users/tibod/Documents/projects/Bos55/bos55-nix-config-cicd/test/vm-test.nix)
NixOS has built-in VM testing via `nixos-rebuild build-vm` and the NixOS test framework. The approach:
1. **Build a VM from any host config**:
```bash
nix build .#nixosConfigurations.Testing.config.system.build.vm
./result/bin/run-Testing-vm
```
2. **NixOS integration test** (`test/vm-test.nix`):
- Spins up a minimal VM cluster (e.g., two nodes)
- Runs deploy-rs against one VM from the other
- Validates activation, rollback, and connectivity
- Uses `nixos-testing` framework (Python test driver)
3. **Full CI pipeline test locally with `act`**:
```bash
# Run the GitHub Actions workflow locally using act
act push --container-architecture linux/amd64
```
> [!NOTE]
> The existing `build.yml` already uses `catthehacker/ubuntu:act-24.04` containers, suggesting `act` is already part of the workflow. VM tests don't require actual network access to target hosts.
---
## Verification Plan
### Automated Tests
- `nix flake check` — validates flake + deployChecks schema
- `nix build .#nixosConfigurations.<host>.config.system.build.toplevel` for each host
- NixOS VM integration test (`test/vm-test.nix`)
### Manual Verification (guinea pig: `Development` or `Testing`)
1. Push to `test-Development` → verify deploy-rs runs `switch-to-configuration test` on 192.168.0.91
2. Reboot `Development` → verify it falls back to previous generation (test branch behavior)
3. Merge to `main` → verify deploy-rs deploys to all enabled hosts with `switch`
4. Intentionally break a config → verify magic rollback activates
5. Push to Harmonia cache → verify another host can pull the closure
6. Check monitoring metrics show correct generation numbers
View git blame Copy permalink