Bos55/nix-config
2
Fork 1
Code Issues 2 Pull requests Projects Releases Packages Wiki Activity
nix-config/DISASTER_RECOVERY.md

2.6 KiB
Raw Permalink Blame History

Disaster Recovery Plan (Zero-Login Strategy)

In the event of a catastrophic failure, this document outlines how to restore services without causing configuration drift (i.e., without SSHing directly into target nodes and making manual undocumented changes).

1. Full Proxmox Host Freeze (Hardware/GPU Bug)

If the entire Proxmox hypervisor freezes (often caused by the AMD GPU reset bug when a VM reboots):

  • DO NOT attempt to SSH into the host or the VMs.
  • Recovery Action: Use your Out-of-Band Management (OOBM) solution. Access your PiKVM, IPMI interface, or remotely controlled Smart Plug to perform a hard power cycle of the host.
  • Upon reboot, Proxmox will automatically start the VMs, and comin will pull the latest working configuration.

2. Production Service Failure (Code Bug)

If a bad PR was merged and a stateless service is failing to start:

  • DO NOT SSH into the VM to fix the config manually.
  • Recovery Action: Open the Forgejo Web UI and click Revert on the merged PR.
  • Within 60 seconds, the comin agent running on the target VM will detect the new commit on the v2 branch, pull the reverted configuration, and restart the service automatically.

3. Production Data Corruption (Database/Stateful Bug)

If a bad deployment corrupted persistent data on TrueNAS (e.g., a failed one-way database schema migration):

  • Recovery Action:
    1. Revert the code PR in Forgejo as described in section 2.
    2. Execute the scripts/dr-rollback-dataset.sh script from the secure Control Center VM. This script uses the TrueNAS REST API to rollback the specific ZFS dataset to the automated snapshot taken immediately prior to the deployment.

4. Total Git Repository Loss

If the v2 branch is completely destroyed or the Gitea server is unrecoverable:

  • Your prod-master private age key is stored safely offline on your USB drive.
  • A recent backup of the repository should be available via your external backup mechanism.
  • Clone the backup to your local workstation, run the proxmox-bootstrap Terraform state locally to ensure the hypervisor is reachable, and use Terraform to redeploy the Control Center VM and Forgejo instances.

5. Manually Obtaining the Golden Image

If the CI pipeline is down and you urgently need to provision a new VM:

  1. Run nix run github:nix-community/nixos-generators -- --format proxmox -c hosts/Template/default.nix -o result on your local machine.
  2. The output .qcow2 image can be uploaded to the Proxmox Web UI manually under local -> ISO Images (or directly via qm importdisk if you must use SSH as a last resort).