# **Infrastructure Specification: Automated NixOS GitOps Environment**

## **1\. Architecture Overview**

This specification outlines the migration from a manually deployed NixOS environment to a fully automated, pull-based GitOps architecture.

* **Source of Truth:** Forgejo repository (v2 branch).  
* **Infrastructure Provisioning:** Terraform (via bpg/proxmox provider) executed by Forgejo Actions.  
* **Configuration Management:** NixOS Flakes, pulled autonomously by individual nodes via comin.  
* **Secret Management:** sops-nix using Age keys (SSH host keys for VM decryption).  
* **Storage:** TrueNAS serving persistent data via NFS/iSCSI, utilizing ZFS snapshot cloning for the staging environment.  
* **Dependency Management:** RenovateBot running via Forgejo Actions.

## **2\. Phase 1: Foundation & State Preparation**

### **2.1 Git Branching Strategy**

* Initialize a new orphan branch named v2 (or a fresh repository) to ensure a clean slate for the flake lockfile and SOPS configuration.  
* Cherry-pick existing .nix service modules from the legacy main branch as they are migrated.

### **2.2 Secret Management (Bankruptcy & Reset)**

* Delete all legacy .sops.yaml configurations.  
* Generate a new master age key. This key will be stored securely offline (e.g., Bitwarden) and **never** placed on the Forgejo runner or target VMs.  
* Initialize a new .sops.yaml. Target VMs will be added via their public SSH host keys as they are provisioned.

### **2.3 Storage Strategy (TrueNAS & Staging)**

To provide the Staging VM with production-accurate data without risking corruption:

1. **Production State:** TrueNAS maintains the primary ZFS datasets.  
2. **Staging State (On-Demand):** When the staging environment spins up, a script (via Proxmox/Forgejo Action or TrueNAS API) takes a temporary ZFS snapshot of the production dataset.  
3. **Clone & Mount:** The snapshot is cloned and exported via NFS/SMB specifically for the Staging VM.  
4. **Teardown:** When Staging is spun down, the ZFS clone is destroyed.

## **3\. Phase 2: Proxmox Configuration & Least Privilege**

Terraform requires API access to Proxmox. To adhere to the principle of least privilege, Proxmox RBAC (Role-Based Access Control) will be utilized.

### **3.1 Proxmox Static Configuration (RBAC)**

The root user (root@pam) must **never** be used for automation. Instead, a dedicated API user with a highly restricted role must be created. This can be done via the Proxmox Web UI (Datacenter \-\> Permissions) or via the Proxmox shell (pveum).  
**1\. Create the Restricted Role (TerraformProv):**  
This role grants only the permissions needed to clone templates, configure hardware, and manage power states.  
pveum role add TerraformProv \-privs "VM.Allocate VM.Audit VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Monitor VM.PowerMgmt Datastore.AllocateSpace Datastore.Audit SDN.Use"

**2\. Create the Terraform User:**  
Create a user in the Proxmox VE authentication realm (@pve).  
pveum user add terraform@pve

**3\. Assign Permissions to Paths:**  
Bind the role to the user, strictly limiting access to the VMs path and the specific storage path where VMs will reside (e.g., local-zfs).  
pveum acl modify /vms \-user terraform@pve \-role TerraformProv  
pveum acl modify /storage/local-zfs \-user terraform@pve \-role TerraformProv

**4\. Generate API Token:**  
Generate a token for the user. Proxmox will output a Secret ID. This ID is saved into Forgejo Secrets and passed to the Terraform provider.  
pveum user token add terraform@pve tf-automation \--privsep 0

*(Note: \--privsep 0 ensures the token inherits the user's permissions, rather than requiring separate ACL definitions).*

## **4\. Phase 3: Infrastructure as Code (Terraform)**

### **4.1 The Golden Image (.qcow2)**

A minimal NixOS image is built locally using nixos-generators and uploaded to Proxmox as a template (e.g., ID 9000). This template contains:

* QEMU Guest Agent.  
* Cloud-Init support.  
* comin installed.

### **4.2 Terraform Provisioning**

Terraform maintains the hardware state. Its responsibilities are strictly limited to:

* Cloning Template 9000\.  
* Allocating vCPU, RAM, and Virtual Disks.  
* Using Cloud-Init to inject network configurations (IP, Gateway) and the Forgejo runner's public SSH key for health check access.

## **5\. Phase 4: CI/CD & Automation**

### **5.1 Forgejo Runner & Terraform Pipeline**

* The newly provisioned Forgejo Runner VM registers with Forgejo.  
* A workflow (terraform.yml) runs terraform plan on Pull Requests and terraform apply on merges to the main branch.

### **5.2 GitOps Pull Agent (comin and Alternatives)**

Once provisioned by Terraform, VMs boot and pull their OS configuration autonomously.

* **Chosen Tool (comin):** Polling agent designed specifically for NixOS GitOps. Runs as a systemd service, supports multiple git remotes, branch tracking, and local cryptographic verification.  
* **Alternative 1 (system.autoUpgrade):** The native NixOS approach utilizing systemd timers. *Rejected* because it lacks advanced Git authentication and webhooks for immediate triggering.  
* **Alternative 2 (deploy-rs / colmena):** Excellent push-based deployment tools. *Rejected* because they require the CI runner to maintain SSH root access to the entire fleet, violating our zero-trust/pull-only goals.

### **5.3 RenovateBot Integration**

* Renovate is deployed via a Forgejo Action running on a cron schedule.  
* Nix flake support is explicitly enabled in renovate.json.  
* Renovate parses flake.lock and Docker tags, automatically opening PRs for updates.

## **6\. Phase 5: Staging Lifecycle & Healthchecks**

### **6.1 Lifecycle Automation**

1. **Trigger:** A PR is opened by Renovate or a developer.  
2. **Wake-up:** A Forgejo Action calls the Proxmox API to start the nixos-staging VM and triggers the TrueNAS ZFS snapshot clone.  
3. **Pull:** The Staging VM boots, connects to the network, and comin pulls the PR's commit hash.

### **6.2 Automated Testing**

The Forgejo Action will verify the deployment was successful:

* **Systemd Checks:** systemctl is-system-running \--wait and systemctl is-active \<service\>.  
* **Docker Healthchecks:** docker inspect \--format='{{json .State.Health.Status}}' \<container\>.  
* **HTTP Probes:** Execute curl commands against the Staging VM to verify HTTP 200 OK.

### **6.3 Spin Down**

Upon PR merge or closure, the Forgejo Action shuts down the VM via the Proxmox API and destroys the temporary TrueNAS ZFS clone.

## **7\. Phase 6: Rollbacks & Disaster Recovery**

Because infrastructure involves both *stateless* configurations (NixOS) and *stateful* data (TrueNAS), a unified rollback strategy is critical.

### **7.1 Differentiating Failures (Nix vs. TrueNAS)**

If a deployment fails, the root cause must be identified to determine the rollback path:

* **Stateless Failure (NixOS/Code):** The service fails to start immediately. System logs (journalctl \-u my-service) show syntax errors, missing binaries, or bad systemd unit definitions.  
* **Stateful Failure (TrueNAS/Data):** The service starts but crashes with a database schema version mismatch, "permission denied" on the NFS mount, or missing user data. This means a service (like a Docker container) attempted a one-way database migration on the persistent TrueNAS dataset and failed halfway.

### **7.2 The Unified Rollback Procedure**

When a production rollout fails, **do not** attempt to fix it live.

1. **Revert the State (TrueNAS):** If the failure was stateful (e.g., a bad database migration), immediately log into TrueNAS and rollback the primary dataset to the automated ZFS snapshot taken right before the deployment.  
2. **Revert the Code (Forgejo):** Use the Forgejo UI to click "Revert" on the problematic Pull Request. This creates a new commit restoring the previous flake.nix state.  
3. **Autonomous Recovery:** Within 60 seconds, the production VM's comin agent will detect the new commit on main, pull the reverted code, apply the old configuration, and reconnect to the restored TrueNAS dataset.