docs: remove apt repo fixes and rely on proxmox helper script

BOOTSTRAP.md

@@ -1,32 +1,26 @@
 # NixOS GitOps Bootstrap Guide
 
-This repository is designed to be fully automated once bootstrapped, but if you are adapting this codebase for **your own infrastructure**, you must modify several deployment-specific variables before running the bootstrap script on a fresh Proxmox host.
+This repository is designed to be fully automated once bootstrapped. We separate concerns into two layers:
+1. **Host Layer**: Bare-metal hardware setup on Proxmox.
+2. **Workload Layer**: VMs and network resources.
 
-## Adapt the Codebase
+## 1. Apply the Host Layer
 
-Before bootstrapping your host, fork or clone this repository and make the following changes to match your environment:
+Before deploying VMs, you need to prepare the Proxmox host.
 
-### Hardware Identifiers
-- **Find your NVMe/Disk UUID**: Log into your fresh Proxmox host and run:
+1. **Run Proxmox Post-Install Script**: Log into your fresh Proxmox node's shell and run the community `proxmox-ve-helper` post-install script to fix the APT repositories and remove the nag screen:
    ```bash
-   ls -l /dev/disk/by-id/
+   bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/misc/post-pve-install.sh)"
    ```
-   Identify your primary data disk (e.g. `nvme-eui...` or `wwn-0x...`).
-- **Update OpenTofu Config**: Open `opentofu/nodes/mikoshi/main.tf` (you may want to rename `mikoshi` to your host's name) and replace the `disk` ID inside the `zpool` resource with your hardware UUID.
+2. Clone this repository to your laptop.
+3. Run the host-layer apply script using Docker (requires Docker installed). The script will automatically configure the remaining hardware-specific requirements (like the ZFS pool, NIC offloading for stability, and the laptop lid switch):
+   ```bash
+   ./scripts/apply-host-layer.sh
+   ```
+4. OpenTofu will prompt you for variables like the target `node_ip`, your `ssh_user`, and the `data_disk_id` (e.g. `nvme-eui...`) to format as ZFS. 
 
-### Identity & Access
-- **SSH Keys**: The GitOps Control Center needs an SSH key for disaster recovery. 
-   - Update the Cloud-Init SSH key in `opentofu/nodes/mikoshi/main.tf` under the `user_account` block.
-   - Update the permanent NixOS SSH key in `nixos/users/admin/default.nix`.
-- **Secrets (SOPS)**: Replace the placeholder tokens in the Control Center host config (e.g., `nixos/hosts/izanagi/secrets.yaml` if you haven't renamed it) with your actual Proxmox API token and Forgejo token. Encrypt this file with your own `sops` Age key.
+## 2. Prepare the Golden Image
 
-### Hostname & Naming Schema
-If your Proxmox host or your Control Center has a different name:
-- Rename the folders in `opentofu/nodes/` and `nixos/hosts/`.
-- Update the `node_name` inside your OpenTofu `main.tf`.
-- Update `nixos/flake.nix` to reflect your new host names.
-
-### The Golden Image
 Because the GitOps Control Center must be spun up fully configured without human intervention, you need a pre-built NixOS `.qcow2` image.
 1. Build the golden image locally (requires Nix/WSL):
    ```bash
@@ -36,25 +30,22 @@ Because the GitOps Control Center must be spun up fully configured without human
 3. Create a new VM in Proxmox with **ID 9000**.
 4. Import the `.qcow2` as its disk and convert the VM into a **Template**. *(Ensure the template has Cloud-Init configured).*
 
-## Execute the Bootstrap
+## 3. Deploy the Workload Layer
 
-Once you have pushed your adapted codebase to your Git server, SSH into your fresh Proxmox node as `root` and run the bootstrap script:
+Once the host is prepped and the template exists, you can deploy the base workloads (like the GitOps Control Center).
 
-```bash
-curl -fsSL https://git.your-server.com/your-repo/raw/branch/main/scripts/bootstrap.sh | bash
-```
-
-### What this script does automatically:
-- **Fixes APT Repositories**: Disables enterprise repositories and adds community repositories.
-- **Fixes NIC Offloading**: Installs a systemd service to safely disable TSO/GSO/GRO on physical interfaces to prevent network drops.
-- **Installs OpenTofu**: Pulls the official binaries.
-- **Applies Host State**: Runs `tofu apply` which:
-   - Formats your specified disk into the `data` ZFS pool.
-   - Sets laptop lid switch to ignore (if applicable).
-   - Spins up the GitOps Control Center VM.
+1. Change to the workload-layer directory:
+   ```bash
+   cd opentofu/workload-layer/production
+   ```
+2. Initialize and apply:
+   ```bash
+   tofu init
+   tofu apply
+   ```
 
 ## Post-Bootstrap
 
-Once the bootstrap script completes, the Control Center VM will boot, initialize via Cloud-Init, and automatically start pulling this git repository.
+Once the `tofu apply` completes, the Control Center VM will boot, initialize via Cloud-Init, and automatically start pulling this git repository.
 
-From this point on, **you no longer need to log into the Proxmox host.** All future changes to VMs, networks, or applications should be done declaratively via Pull Requests to your repository!
+From this point on, **you no longer need to manually run tofu apply.** All future changes to VMs, networks, or applications should be done declaratively via Pull Requests to your repository!

nixos/modules/services/hypervisor-gitops/default.nix

@@ -80,7 +80,7 @@ in {
         StateDirectory = "hypervisor-gitops";
         WorkingDirectory = "/var/lib/hypervisor-gitops";
         
-        ExecStart = "${hypervisorSyncScript}/bin/hypervisor-sync ${cfg.repoUrl} opentofu/nodes/mikoshi";
+        ExecStart = "${hypervisorSyncScript}/bin/hypervisor-sync ${cfg.repoUrl} opentofu/workload-layer/production";
       };
     };
 

opentofu/host-layer/main.tf

@@ -0,0 +1,54 @@
+terraform {
+  # This state is dedicated purely to the physical host configuration.
+  # No providers are required since we use null_resource and ssh.
+}
+
+resource "null_resource" "bare_metal_setup" {
+  triggers = {
+    node         = var.node_ip
+    data_disk_id = var.data_disk_id
+    pool_name    = var.zfs_pool_name
+  }
+
+  connection {
+    type        = "ssh"
+    user        = var.ssh_user
+    host        = var.node_ip
+    private_key = var.ssh_private_key != "" ? var.ssh_private_key : null
+    agent       = var.ssh_private_key == "" ? true : false
+  }
+
+  provisioner "remote-exec" {
+    inline = [
+      "set -euo pipefail",
+
+
+      "echo '==> Setting laptop lid switch to ignore (prevents sleeping when closed)...'",
+      "sed -i 's/^#\\?HandleLidSwitch=.*/HandleLidSwitch=ignore/' /etc/systemd/logind.conf",
+      "systemctl restart systemd-logind",
+      
+      "echo '==> Applying NIC offloading fixes...'",
+      "cat <<'EOF' > /etc/systemd/system/nic-offload-fix.service",
+      "[Unit]",
+      "Description=Disable NIC offloading (TSO/GRO/GSO) for physical interfaces",
+      "After=network-online.target",
+      "",
+      "[Service]",
+      "Type=oneshot",
+      "ExecStart=/bin/bash -c 'for dev in /sys/class/net/*; do if [ \"$(basename \"$dev\")\" != \"lo\" ] && [[ ! \"$(basename \"$dev\")\" =~ ^(vmbr|veth|fwbr|tap|bonding) ]]; then /usr/sbin/ethtool -K \"$(basename \"$dev\")\" tso off gso off gro off || true; fi; done'",
+      "RemainAfterExit=yes",
+      "",
+      "[Install]",
+      "WantedBy=multi-user.target",
+      "EOF",
+      "systemctl daemon-reload",
+      "systemctl enable --now nic-offload-fix.service || true",
+
+      "echo '==> Setting up ZFS pool and Proxmox storage...'",
+      "zpool list ${var.zfs_pool_name} || zpool create -f ${var.zfs_pool_name} /dev/disk/by-id/${var.data_disk_id}",
+      "pvesm status -storage ${var.zfs_pool_name} || pvesm add zfspool ${var.zfs_pool_name} --pool ${var.zfs_pool_name} --content images,rootdir",
+
+      "echo '==> Host layer configuration complete!'"
+    ]
+  }
+}

opentofu/host-layer/variables.tf

@@ -0,0 +1,27 @@
+variable "node_ip" {
+  type        = string
+  description = "The IP address or hostname of the Proxmox node to configure"
+}
+
+variable "ssh_user" {
+  type        = string
+  default     = "root"
+  description = "The SSH user to connect as"
+}
+
+variable "ssh_private_key" {
+  type        = string
+  default     = ""
+  description = "The SSH private key content (if not using ssh-agent)"
+}
+
+variable "data_disk_id" {
+  type        = string
+  description = "The persistent block device ID for the data disk (e.g. wwn-0x500...)"
+}
+
+variable "zfs_pool_name" {
+  type        = string
+  default     = "data"
+  description = "The name of the ZFS pool to create on the data disk"
+}

opentofu/modules/proxmox-node/main.tf

@@ -7,40 +7,7 @@ terraform {
   }
 }
 
-# Bare Metal Host Configurations (SSH Provisioning)
 
-# We use a null_resource to run imperative commands on the Debian host
-# that are not currently supported by the bpg/proxmox provider.
-resource "null_resource" "bare_metal_setup" {
-  triggers = {
-    node         = var.node_name
-    data_disk_id = var.data_disk_id
-    pool_name    = var.zfs_pool_name
-  }
-
-  connection {
-    type     = "ssh"
-    user     = "root"
-    # Assuming running locally on the node during bootstrap, or via SSH if run from a laptop.
-    # We default to local host if run from Control Center, but for flexibility we use the endpoint.
-    host     = var.node_name 
-    agent    = true
-  }
-
-  provisioner "remote-exec" {
-    inline = [
-      # Set laptop lid switch to ignore (prevents sleeping when closed)
-      "sed -i 's/^#\\?HandleLidSwitch=.*/HandleLidSwitch=ignore/' /etc/systemd/logind.conf",
-      "systemctl restart systemd-logind",
-      
-      # Format the ZFS pool if it doesn't already exist
-      "zpool list ${var.zfs_pool_name} || zpool create -f ${var.zfs_pool_name} /dev/disk/by-id/${var.data_disk_id}",
-      
-      # Register the ZFS pool in Proxmox if it's not already registered
-      "pvesm status -storage ${var.zfs_pool_name} || pvesm add zfspool ${var.zfs_pool_name} --pool ${var.zfs_pool_name} --content images,rootdir"
-    ]
-  }
-}
 
 # Resource Pools
 

opentofu/modules/proxmox-node/variables.tf

@@ -3,10 +3,6 @@ variable "node_name" {
   description = "The name of the Proxmox node (e.g. pve)"
 }
 
-variable "data_disk_id" {
-  type        = string
-  description = "The persistent block device ID for the data disk (e.g. wwn-0x500...)"
-}
 
 variable "zfs_pool_name" {
   type        = string

opentofu/workload-layer/production/main.tf

@@ -19,7 +19,6 @@ provider "proxmox" {
 module "proxmox_node" {
   source = "../../modules/proxmox-node"
   node_name = "mikoshi"
-  data_disk_id = "nvme-KXG80ZNV2T04_NVMe_KIOXIA_2048GB_241C11Y5EHAK"
   zfs_pool_name = "data"
 }
 

scripts/apply-host-layer.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# This script runs the host-layer OpenTofu configuration using a Docker container,
+# keeping your laptop and the Proxmox node clean of dependencies.
+
+echo "Running OpenTofu for host-layer..."
+
+docker run --rm -it \
+  -v "$(pwd)":/workspace \
+  -w /workspace/opentofu/host-layer \
+  -v "$HOME/.ssh:/root/.ssh:ro" \
+  ghcr.io/opentofu/opentofu:latest init -upgrade
+
+docker run --rm -it \
+  -v "$(pwd)":/workspace \
+  -w /workspace/opentofu/host-layer \
+  -v "$HOME/.ssh:/root/.ssh:ro" \
+  ghcr.io/opentofu/opentofu:latest apply

scripts/bootstrap.sh

@@ -1,76 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-BANNER="==========================================================="
-printf "%s\n Proxmox Bootstrap\n%s\n\n" "$BANNER" "$BANNER"
-
-echo "Applying post-pve-install fixes (fixing repos)..."
-# Remove enterprise repos and add non-subscription repos safely
-rm -f /etc/apt/sources.list.d/pve-enterprise.list
-
-echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list
-
-# Disable the "No Valid Subscription" nag screen
-sed -i.bak "s/data.status !== 'Active'/false/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
-
-echo ""
-echo "Applying NIC offloading fixes..."
-# The community script disables TSO, GSO, and GRO on physical interfaces.
-# We create a systemd service to ensure this applies on boot.
-cat <<'EOF' > /etc/systemd/system/nic-offload-fix.service
-[Unit]
-Description=Disable NIC offloading (TSO/GRO/GSO) for physical interfaces
-After=network-online.target
-
-[Service]
-Type=oneshot
-# Iterate over all physical interfaces (excluding lo, bridges, veth, etc.)
-ExecStart=/bin/bash -c 'for dev in /sys/class/net/*; do if [ "$(basename "$dev")" != "lo" ] && [[ ! "$(basename "$dev")" =~ ^(vmbr|veth|fwbr|tap|bonding) ]]; then /usr/sbin/ethtool -K "$(basename "$dev")" tso off gso off gro off || true; fi; done'
-RemainAfterExit=yes
-
-[Install]
-WantedBy=multi-user.target
-EOF
-systemctl enable --now nic-offload-fix.service || true
-
-echo ""
-echo "Updating system and installing OpenTofu..."
-apt-get update
-# Install curl, git, gnupg, ethtool, and required apt dependencies
-apt-get install -y apt-transport-https ca-certificates curl git gnupg ethtool
-
-# Install OpenTofu repository and binary
-install -m 0755 -d /etc/apt/keyrings
-curl -fsSL https://get.opentofu.org/opentofu.gpg | tee /etc/apt/keyrings/opentofu.gpg >/dev/null
-curl -fsSL https://packages.opentofu.org/opentofu/tofu/gpgkey | gpg --no-tty --batch --dearmor -o /etc/apt/keyrings/opentofu-repo.gpg >/dev/null
-chmod a+r /etc/apt/keyrings/opentofu.gpg /etc/apt/keyrings/opentofu-repo.gpg
-
-printf "deb [signed-by=/etc/apt/keyrings/opentofu.gpg,/etc/apt/keyrings/opentofu-repo.gpg] https://packages.opentofu.org/opentofu/tofu/any/ any main\ndeb-src [signed-by=/etc/apt/keyrings/opentofu.gpg,/etc/apt/keyrings/opentofu-repo.gpg] https://packages.opentofu.org/opentofu/tofu/any/ any main\n" > /etc/apt/sources.list.d/opentofu.list
-chmod a+r /etc/apt/sources.list.d/opentofu.list
-
-apt-get update
-apt-get install -y tofu
-
-echo ""
-echo "Cloning the infrastructure repository..."
-cd /root
-if [ -d "nix-config" ]; then
-    echo "Repository already exists. Pulling latest..."
-    cd nix-config
-    git pull
-else
-    git clone -b v2 https://git.depeuter.dev/Bos55/nix-config.git
-    cd nix-config
-fi
-
-echo ""
-echo "Bootstrapping Proxmox Host State..."
-cd opentofu/nodes/mikoshi
-
-echo "Initializing OpenTofu..."
-tofu init -upgrade
-
-echo "Applying bare-metal state..."
-tofu apply -auto-approve
-
-printf "\n%s\n Bootstrap Complete!\n The ZFS pool, Resource Pools, and the Control Center\n VM have been provisioned.\n\n The Control Center is booting now. Once online, it will automatically\n pull this repository and provision the rest of your VMs!\n%s\n" "$BANNER" "$BANNER"

scripts/hypervisor-sync.sh

@@ -5,7 +5,7 @@
 # Usage: ./hypervisor-sync.sh <REPO_URL> <OPENTOFU_DIR>
 
 REPO_URL=${1:-"https://git.depeuter.dev/Bos55/nix-config.git"}
-OPENTOFU_DIR=${2:-"opentofu/nodes/mikoshi"}
+OPENTOFU_DIR=${2:-"opentofu/workload-layer/production"}
 
 echo "Starting Hypervisor GitOps sync..."
 

scripts/staging-sync.sh

@@ -26,7 +26,7 @@ fi
 
 # Ensure workspace directory exists for this PR
 WORKSPACE="/var/lib/hypervisor-gitops/staging-pr-${PR_NUMBER}"
-OPENTOFU_SRC="/var/lib/hypervisor-gitops/nix-config/opentofu/staging-env"
+OPENTOFU_SRC="/var/lib/hypervisor-gitops/nix-config/opentofu/workload-layer/staging"
 
 # TrueNAS API variables
 TRUENAS_URL=${TRUENAS_URL:-"https://192.168.0.11"}