32 lines
2.6 KiB
Markdown
32 lines
2.6 KiB
Markdown
# Disaster Recovery Plan (Zero-Login Strategy)
|
|
|
|
In the event of a catastrophic failure, this document outlines how to restore services without causing configuration drift (i.e., without SSHing directly into target nodes and making manual undocumented changes).
|
|
|
|
## 1. Full Proxmox Host Freeze (Hardware/GPU Bug)
|
|
If the entire Proxmox hypervisor freezes (often caused by the AMD GPU reset bug when a VM reboots):
|
|
* **DO NOT** attempt to SSH into the host or the VMs.
|
|
* **Recovery Action**: Use your Out-of-Band Management (OOBM) solution. Access your PiKVM, IPMI interface, or remotely controlled Smart Plug to perform a **hard power cycle** of the host.
|
|
* Upon reboot, Proxmox will automatically start the VMs, and `comin` will pull the latest working configuration.
|
|
|
|
## 2. Production Service Failure (Code Bug)
|
|
If a bad PR was merged and a stateless service is failing to start:
|
|
* **DO NOT** SSH into the VM to fix the config manually.
|
|
* **Recovery Action**: Open the Forgejo Web UI and click **Revert** on the merged PR.
|
|
* Within 60 seconds, the `comin` agent running on the target VM will detect the new commit on the `v2` branch, pull the reverted configuration, and restart the service automatically.
|
|
|
|
## 3. Production Data Corruption (Database/Stateful Bug)
|
|
If a bad deployment corrupted persistent data on TrueNAS (e.g., a failed one-way database schema migration):
|
|
* **Recovery Action**:
|
|
1. Revert the code PR in Forgejo as described in section 2.
|
|
2. Execute the `scripts/dr-rollback-dataset.sh` script from the secure **Control Center VM**. This script uses the TrueNAS REST API to rollback the specific ZFS dataset to the automated snapshot taken immediately prior to the deployment.
|
|
|
|
## 4. Total Git Repository Loss
|
|
If the `v2` branch is completely destroyed or the Gitea server is unrecoverable:
|
|
* Your `prod-master` private age key is stored safely offline on your USB drive.
|
|
* A recent backup of the repository should be available via your external backup mechanism.
|
|
* Clone the backup to your local workstation, run the `proxmox-bootstrap` Terraform state locally to ensure the hypervisor is reachable, and use Terraform to redeploy the Control Center VM and Forgejo instances.
|
|
|
|
## 5. Manually Obtaining the Golden Image
|
|
If the CI pipeline is down and you urgently need to provision a new VM:
|
|
1. Run `nix run github:nix-community/nixos-generators -- --format proxmox -c hosts/Template/default.nix -o result` on your local machine.
|
|
2. The output `.qcow2` image can be uploaded to the Proxmox Web UI manually under `local` -> `ISO Images` (or directly via `qm importdisk` if you must use SSH as a last resort).
|