# Disaster Recovery Plan (Zero-Login Strategy)

In the event of a catastrophic failure, this document outlines how to restore services without causing configuration drift (i.e., without SSHing directly into target nodes and making manual undocumented changes).

## 1. Full Proxmox Host Freeze (Hardware/GPU Bug)
If the entire Proxmox hypervisor freezes (often caused by the AMD GPU reset bug when a VM reboots):
*   **DO NOT** attempt to SSH into the host or the VMs.
*   **Recovery Action**: Use your Out-of-Band Management (OOBM) solution. Access your PiKVM, IPMI interface, or remotely controlled Smart Plug to perform a **hard power cycle** of the host.
*   Upon reboot, Proxmox will automatically start the VMs, and `comin` will pull the latest working configuration.

## 2. Production Service Failure (Code Bug)
If a bad PR was merged and a stateless service is failing to start:
*   **DO NOT** SSH into the VM to fix the config manually.
*   **Recovery Action**: Open the Forgejo Web UI and click **Revert** on the merged PR.
*   Within 60 seconds, the `comin` agent running on the target VM will detect the new commit on the `v2` branch, pull the reverted configuration, and restart the service automatically.

## 3. Production Data Corruption (Database/Stateful Bug)
If a bad deployment corrupted persistent data on TrueNAS (e.g., a failed one-way database schema migration):
*   **Recovery Action**: 
    1. Revert the code PR in Forgejo as described in section 2.
    2. Execute the `scripts/dr-rollback-dataset.sh` script from the secure **Control Center VM**. This script uses the TrueNAS REST API to rollback the specific ZFS dataset to the automated snapshot taken immediately prior to the deployment.

## 4. Total Git Repository Loss
If the `v2` branch is completely destroyed or the Gitea server is unrecoverable:
*   Your `prod-master` private age key is stored safely offline on your USB drive.
*   A recent backup of the repository should be available via your external backup mechanism.
*   Clone the backup to your local workstation, run the `proxmox-bootstrap` Terraform state locally to ensure the hypervisor is reachable, and use Terraform to redeploy the Control Center VM and Forgejo instances.

## 5. Manually Obtaining the Golden Image
If the CI pipeline is down and you urgently need to provision a new VM:
1. Run `nix run github:nix-community/nixos-generators -- --format proxmox -c hosts/Template/default.nix -o result` on your local machine.
2. The output `.qcow2` image can be uploaded to the Proxmox Web UI manually under `local` -> `ISO Images` (or directly via `qm importdisk` if you must use SSH as a last resort).