nix-config/NixOS GitOps Migration Specification.md

Infrastructure Specification: Automated NixOS GitOps Environment

1. Architecture Overview

This specification outlines the migration from a manually deployed NixOS environment to a fully automated, pull-based GitOps architecture.

• Source of Truth: Forgejo repository (v2 branch).
• Infrastructure Provisioning: Terraform (via bpg/proxmox provider) executed by Forgejo Actions.
• Configuration Management: NixOS Flakes, pulled autonomously by individual nodes via comin.
• Secret Management: sops-nix using Age keys (SSH host keys for VM decryption).
• Storage: TrueNAS serving persistent data via NFS/iSCSI, utilizing ZFS snapshot cloning for the staging environment.
• Dependency Management: RenovateBot running via Forgejo Actions.

2. Phase 1: Foundation & State Preparation

2.1 Git Branching Strategy

• Initialize a new orphan branch named v2 (or a fresh repository) to ensure a clean slate for the flake lockfile and SOPS configuration.
• Cherry-pick existing .nix service modules from the legacy main branch as they are migrated.

2.2 Secret Management (Bankruptcy & Reset)

• Delete all legacy .sops.yaml configurations.
• Generate a new master age key. This key will be stored securely offline (e.g., Bitwarden) and never placed on the Forgejo runner or target VMs.
• Initialize a new .sops.yaml. Target VMs will be added via their public SSH host keys as they are provisioned.

2.3 Storage Strategy (TrueNAS & Staging)

To provide the Staging VM with production-accurate data without risking corruption:

1. Production State: TrueNAS maintains the primary ZFS datasets.
2. Staging State (On-Demand): When the staging environment spins up, a script (via Proxmox/Forgejo Action or TrueNAS API) takes a temporary ZFS snapshot of the production dataset.
3. Clone & Mount: The snapshot is cloned and exported via NFS/SMB specifically for the Staging VM.
4. Teardown: When Staging is spun down, the ZFS clone is destroyed.

3. Phase 2: Proxmox Configuration & Least Privilege

Terraform requires API access to Proxmox. To adhere to the principle of least privilege, Proxmox RBAC (Role-Based Access Control) will be utilized.

3.1 Proxmox Static Configuration (RBAC)

The root user (root@pam) must never be used for automation. Instead, a dedicated API user with a highly restricted role must be created. This can be done via the Proxmox Web UI (Datacenter -> Permissions) or via the Proxmox shell (pveum).
1. Create the Restricted Role (TerraformProv):
This role grants only the permissions needed to clone templates, configure hardware, and manage power states.
pveum role add TerraformProv -privs "VM.Allocate VM.Audit VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Monitor VM.PowerMgmt Datastore.AllocateSpace Datastore.Audit SDN.Use"

2. Create the Terraform User:
Create a user in the Proxmox VE authentication realm (@pve).
pveum user add terraform@pve

3. Assign Permissions to Paths:
Bind the role to the user, strictly limiting access to the VMs path and the specific storage path where VMs will reside (e.g., local-zfs).
pveum acl modify /vms -user terraform@pve -role TerraformProv
pveum acl modify /storage/local-zfs -user terraform@pve -role TerraformProv

4. Generate API Token:
Generate a token for the user. Proxmox will output a Secret ID. This ID is saved into Forgejo Secrets and passed to the Terraform provider.
pveum user token add terraform@pve tf-automation --privsep 0

(Note: --privsep 0 ensures the token inherits the user's permissions, rather than requiring separate ACL definitions).

4. Phase 3: Infrastructure as Code (Terraform)

4.1 The Golden Image (.qcow2)

A minimal NixOS image is built locally using nixos-generators and uploaded to Proxmox as a template (e.g., ID 9000). This template contains:

• QEMU Guest Agent.
• Cloud-Init support.
• comin installed.

4.2 Terraform Provisioning

Terraform maintains the hardware state. Its responsibilities are strictly limited to:

• Cloning Template 9000.
• Allocating vCPU, RAM, and Virtual Disks.
• Using Cloud-Init to inject network configurations (IP, Gateway) and the Forgejo runner's public SSH key for health check access.

5. Phase 4: CI/CD & Automation

5.1 Forgejo Runner & Terraform Pipeline

• The newly provisioned Forgejo Runner VM registers with Forgejo.
• A workflow (terraform.yml) runs terraform plan on Pull Requests and terraform apply on merges to the main branch.

5.2 GitOps Pull Agent (comin and Alternatives)

Once provisioned by Terraform, VMs boot and pull their OS configuration autonomously.

• Chosen Tool (comin): Polling agent designed specifically for NixOS GitOps. Runs as a systemd service, supports multiple git remotes, branch tracking, and local cryptographic verification.
• Alternative 1 (system.autoUpgrade): The native NixOS approach utilizing systemd timers. Rejected because it lacks advanced Git authentication and webhooks for immediate triggering.
• Alternative 2 (deploy-rs / colmena): Excellent push-based deployment tools. Rejected because they require the CI runner to maintain SSH root access to the entire fleet, violating our zero-trust/pull-only goals.

5.3 RenovateBot Integration

• Renovate is deployed via a Forgejo Action running on a cron schedule.
• Nix flake support is explicitly enabled in renovate.json.
• Renovate parses flake.lock and Docker tags, automatically opening PRs for updates.

6. Phase 5: Staging Lifecycle & Healthchecks

6.1 Lifecycle Automation

1. Trigger: A PR is opened by Renovate or a developer.
2. Wake-up: A Forgejo Action calls the Proxmox API to start the nixos-staging VM and triggers the TrueNAS ZFS snapshot clone.
3. Pull: The Staging VM boots, connects to the network, and comin pulls the PR's commit hash.

6.2 Automated Testing

The Forgejo Action will verify the deployment was successful:

• Systemd Checks: systemctl is-system-running --wait and systemctl is-active <service>.
• Docker Healthchecks: docker inspect --format='{{json .State.Health.Status}}' <container>.
• HTTP Probes: Execute curl commands against the Staging VM to verify HTTP 200 OK.

6.3 Spin Down

Upon PR merge or closure, the Forgejo Action shuts down the VM via the Proxmox API and destroys the temporary TrueNAS ZFS clone.

7. Phase 6: Rollbacks & Disaster Recovery

Because infrastructure involves both stateless configurations (NixOS) and stateful data (TrueNAS), a unified rollback strategy is critical.

7.1 Differentiating Failures (Nix vs. TrueNAS)

If a deployment fails, the root cause must be identified to determine the rollback path:

• Stateless Failure (NixOS/Code): The service fails to start immediately. System logs (journalctl -u my-service) show syntax errors, missing binaries, or bad systemd unit definitions.
• Stateful Failure (TrueNAS/Data): The service starts but crashes with a database schema version mismatch, "permission denied" on the NFS mount, or missing user data. This means a service (like a Docker container) attempted a one-way database migration on the persistent TrueNAS dataset and failed halfway.

7.2 The Unified Rollback Procedure

When a production rollout fails, do not attempt to fix it live.

1. Revert the State (TrueNAS): If the failure was stateful (e.g., a bad database migration), immediately log into TrueNAS and rollback the primary dataset to the automated ZFS snapshot taken right before the deployment.
2. Revert the Code (Forgejo): Use the Forgejo UI to click "Revert" on the problematic Pull Request. This creates a new commit restoring the previous flake.nix state.
3. Autonomous Recovery: Within 60 seconds, the production VM's comin agent will detect the new commit on main, pull the reverted code, apply the old configuration, and reconnect to the restored TrueNAS dataset.