Bos55/nix-config
2
Fork 1
Code Issues 2 Pull requests Projects Releases Packages Wiki Activity

ci: add golden image build action and document disaster recovery

Browse source
This commit is contained in:
Tibo De Peuter 2026-07-17 22:07:49 +02:00
parent 12b1faf029
commit dd2065cf60
Signed by: tdpeuter
SSH key fingerprint: SHA256:u/h/LVoqKF1Iz02uOyxe6hcjmoZASCGV2HM0TG9ZMoU
2 changed files with 74 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

32
DISASTER_RECOVERY.md Normal file
View file

@ -0,0 +1,32 @@
# Disaster Recovery Plan (Zero-Login Strategy)
In the event of a catastrophic failure, this document outlines how to restore services without causing configuration drift (i.e., without SSHing directly into target nodes and making manual undocumented changes).
## 1. Full Proxmox Host Freeze (Hardware/GPU Bug)
If the entire Proxmox hypervisor freezes (often caused by the AMD GPU reset bug when a VM reboots):
* **DO NOT** attempt to SSH into the host or the VMs.
* **Recovery Action**: Use your Out-of-Band Management (OOBM) solution. Access your PiKVM, IPMI interface, or remotely controlled Smart Plug to perform a **hard power cycle** of the host.
* Upon reboot, Proxmox will automatically start the VMs, and `comin` will pull the latest working configuration.
## 2. Production Service Failure (Code Bug)
If a bad PR was merged and a stateless service is failing to start:
* **DO NOT** SSH into the VM to fix the config manually.
* **Recovery Action**: Open the Forgejo Web UI and click **Revert** on the merged PR.
* Within 60 seconds, the `comin` agent running on the target VM will detect the new commit on the `v2` branch, pull the reverted configuration, and restart the service automatically.
## 3. Production Data Corruption (Database/Stateful Bug)
If a bad deployment corrupted persistent data on TrueNAS (e.g., a failed one-way database schema migration):
* **Recovery Action**:
1. Revert the code PR in Forgejo as described in section 2.
2. Execute the `scripts/dr-rollback-dataset.sh` script from the secure **Control Center VM**. This script uses the TrueNAS REST API to rollback the specific ZFS dataset to the automated snapshot taken immediately prior to the deployment.
## 4. Total Git Repository Loss
If the `v2` branch is completely destroyed or the Gitea server is unrecoverable:
* Your `prod-master` private age key is stored safely offline on your USB drive.
* A recent backup of the repository should be available via your external backup mechanism.
* Clone the backup to your local workstation, run the `proxmox-bootstrap` Terraform state locally to ensure the hypervisor is reachable, and use Terraform to redeploy the Control Center VM and Forgejo instances.
## 5. Manually Obtaining the Golden Image
If the CI pipeline is down and you urgently need to provision a new VM:
1. Run `nix run github:nix-community/nixos-generators -- --format proxmox -c hosts/Template/default.nix -o result` on your local machine.
2. The output `.qcow2` image can be uploaded to the Proxmox Web UI manually under `local` -> `ISO Images` (or directly via `qm importdisk` if you must use SSH as a last resort).