feat(gitops): implement native custom gitops solution with webhooks

NixOS GitOps Migration Specification.md

@@ -0,0 +1,138 @@
+# **Infrastructure Specification: Automated NixOS GitOps Environment**
+
+## **1\. Architecture Overview**
+
+This specification outlines the migration from a manually deployed NixOS environment to a fully automated, pull-based GitOps architecture.
+
+* **Source of Truth:** Forgejo repository (v2 branch).  
+* **Infrastructure Provisioning:** Terraform (via bpg/proxmox provider) executed by Forgejo Actions.  
+* **Configuration Management:** NixOS Flakes, pulled autonomously by individual nodes via comin.  
+* **Secret Management:** sops-nix using Age keys (SSH host keys for VM decryption).  
+* **Storage:** TrueNAS serving persistent data via NFS/iSCSI, utilizing ZFS snapshot cloning for the staging environment.  
+* **Dependency Management:** RenovateBot running via Forgejo Actions.
+
+## **2\. Phase 1: Foundation & State Preparation**
+
+### **2.1 Git Branching Strategy**
+
+* Initialize a new orphan branch named v2 (or a fresh repository) to ensure a clean slate for the flake lockfile and SOPS configuration.  
+* Cherry-pick existing .nix service modules from the legacy main branch as they are migrated.
+
+### **2.2 Secret Management (Bankruptcy & Reset)**
+
+* Delete all legacy .sops.yaml configurations.  
+* Generate a new master age key. This key will be stored securely offline (e.g., Bitwarden) and **never** placed on the Forgejo runner or target VMs.  
+* Initialize a new .sops.yaml. Target VMs will be added via their public SSH host keys as they are provisioned.
+
+### **2.3 Storage Strategy (TrueNAS & Staging)**
+
+To provide the Staging VM with production-accurate data without risking corruption:
+
+1. **Production State:** TrueNAS maintains the primary ZFS datasets.  
+2. **Staging State (On-Demand):** When the staging environment spins up, a script (via Proxmox/Forgejo Action or TrueNAS API) takes a temporary ZFS snapshot of the production dataset.  
+3. **Clone & Mount:** The snapshot is cloned and exported via NFS/SMB specifically for the Staging VM.  
+4. **Teardown:** When Staging is spun down, the ZFS clone is destroyed.
+
+## **3\. Phase 2: Proxmox Configuration & Least Privilege**
+
+Terraform requires API access to Proxmox. To adhere to the principle of least privilege, Proxmox RBAC (Role-Based Access Control) will be utilized.
+
+### **3.1 Proxmox Static Configuration (RBAC)**
+
+The root user (root@pam) must **never** be used for automation. Instead, a dedicated API user with a highly restricted role must be created. This can be done via the Proxmox Web UI (Datacenter \-\> Permissions) or via the Proxmox shell (pveum).  
+**1\. Create the Restricted Role (TerraformProv):**  
+This role grants only the permissions needed to clone templates, configure hardware, and manage power states.  
+pveum role add TerraformProv \-privs "VM.Allocate VM.Audit VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Monitor VM.PowerMgmt Datastore.AllocateSpace Datastore.Audit SDN.Use"
+
+**2\. Create the Terraform User:**  
+Create a user in the Proxmox VE authentication realm (@pve).  
+pveum user add terraform@pve
+
+**3\. Assign Permissions to Paths:**  
+Bind the role to the user, strictly limiting access to the VMs path and the specific storage path where VMs will reside (e.g., local-zfs).  
+pveum acl modify /vms \-user terraform@pve \-role TerraformProv  
+pveum acl modify /storage/local-zfs \-user terraform@pve \-role TerraformProv
+
+**4\. Generate API Token:**  
+Generate a token for the user. Proxmox will output a Secret ID. This ID is saved into Forgejo Secrets and passed to the Terraform provider.  
+pveum user token add terraform@pve tf-automation \--privsep 0
+
+*(Note: \--privsep 0 ensures the token inherits the user's permissions, rather than requiring separate ACL definitions).*
+
+## **4\. Phase 3: Infrastructure as Code (Terraform)**
+
+### **4.1 The Golden Image (.qcow2)**
+
+A minimal NixOS image is built locally using nixos-generators and uploaded to Proxmox as a template (e.g., ID 9000). This template contains:
+
+* QEMU Guest Agent.  
+* Cloud-Init support.  
+* comin installed.
+
+### **4.2 Terraform Provisioning**
+
+Terraform maintains the hardware state. Its responsibilities are strictly limited to:
+
+* Cloning Template 9000\.  
+* Allocating vCPU, RAM, and Virtual Disks.  
+* Using Cloud-Init to inject network configurations (IP, Gateway) and the Forgejo runner's public SSH key for health check access.
+
+## **5\. Phase 4: CI/CD & Automation**
+
+### **5.1 Forgejo Runner & Terraform Pipeline**
+
+* The newly provisioned Forgejo Runner VM registers with Forgejo.  
+* A workflow (terraform.yml) runs terraform plan on Pull Requests and terraform apply on merges to the main branch.
+
+### **5.2 GitOps Pull Agent (comin and Alternatives)**
+
+Once provisioned by Terraform, VMs boot and pull their OS configuration autonomously.
+
+* **Chosen Tool (comin):** Polling agent designed specifically for NixOS GitOps. Runs as a systemd service, supports multiple git remotes, branch tracking, and local cryptographic verification.  
+* **Alternative 1 (system.autoUpgrade):** The native NixOS approach utilizing systemd timers. *Rejected* because it lacks advanced Git authentication and webhooks for immediate triggering.  
+* **Alternative 2 (deploy-rs / colmena):** Excellent push-based deployment tools. *Rejected* because they require the CI runner to maintain SSH root access to the entire fleet, violating our zero-trust/pull-only goals.
+
+### **5.3 RenovateBot Integration**
+
+* Renovate is deployed via a Forgejo Action running on a cron schedule.  
+* Nix flake support is explicitly enabled in renovate.json.  
+* Renovate parses flake.lock and Docker tags, automatically opening PRs for updates.
+
+## **6\. Phase 5: Staging Lifecycle & Healthchecks**
+
+### **6.1 Lifecycle Automation**
+
+1. **Trigger:** A PR is opened by Renovate or a developer.  
+2. **Wake-up:** A Forgejo Action calls the Proxmox API to start the nixos-staging VM and triggers the TrueNAS ZFS snapshot clone.  
+3. **Pull:** The Staging VM boots, connects to the network, and comin pulls the PR's commit hash.
+
+### **6.2 Automated Testing**
+
+The Forgejo Action will verify the deployment was successful:
+
+* **Systemd Checks:** systemctl is-system-running \--wait and systemctl is-active \<service\>.  
+* **Docker Healthchecks:** docker inspect \--format='{{json .State.Health.Status}}' \<container\>.  
+* **HTTP Probes:** Execute curl commands against the Staging VM to verify HTTP 200 OK.
+
+### **6.3 Spin Down**
+
+Upon PR merge or closure, the Forgejo Action shuts down the VM via the Proxmox API and destroys the temporary TrueNAS ZFS clone.
+
+## **7\. Phase 6: Rollbacks & Disaster Recovery**
+
+Because infrastructure involves both *stateless* configurations (NixOS) and *stateful* data (TrueNAS), a unified rollback strategy is critical.
+
+### **7.1 Differentiating Failures (Nix vs. TrueNAS)**
+
+If a deployment fails, the root cause must be identified to determine the rollback path:
+
+* **Stateless Failure (NixOS/Code):** The service fails to start immediately. System logs (journalctl \-u my-service) show syntax errors, missing binaries, or bad systemd unit definitions.  
+* **Stateful Failure (TrueNAS/Data):** The service starts but crashes with a database schema version mismatch, "permission denied" on the NFS mount, or missing user data. This means a service (like a Docker container) attempted a one-way database migration on the persistent TrueNAS dataset and failed halfway.
+
+### **7.2 The Unified Rollback Procedure**
+
+When a production rollout fails, **do not** attempt to fix it live.
+
+1. **Revert the State (TrueNAS):** If the failure was stateful (e.g., a bad database migration), immediately log into TrueNAS and rollback the primary dataset to the automated ZFS snapshot taken right before the deployment.  
+2. **Revert the Code (Forgejo):** Use the Forgejo UI to click "Revert" on the problematic Pull Request. This creates a new commit restoring the previous flake.nix state.  
+3. **Autonomous Recovery:** Within 60 seconds, the production VM's comin agent will detect the new commit on main, pull the reverted code, apply the old configuration, and reconnect to the restored TrueNAS dataset.