Compare commits
3 commits
06500a8f01
...
2b533f4375
| Author | SHA1 | Date | |
|---|---|---|---|
2b533f4375
|
|||
|
dc9b7a30d9
|
|||
|
89cec4e0f3
|
15 changed files with 537 additions and 210 deletions
131
BOOTSTRAP.md
131
BOOTSTRAP.md
|
|
@ -1,113 +1,50 @@
|
|||
# Bootstrap Guide
|
||||
# NixOS GitOps Bootstrap Guide
|
||||
|
||||
This document outlines the manual steps required to initialize the NixOS GitOps environment on a fresh Proxmox host. You must perform these steps before the CI/CD pipeline or any automated staging environments can function.
|
||||
This repository is designed to be fully automated once bootstrapped, but if you are adapting this codebase for **your own infrastructure**, you must modify several deployment-specific variables before running the bootstrap script on a fresh Proxmox host.
|
||||
|
||||
## Secret Management Initialization
|
||||
## Adapt the Codebase
|
||||
|
||||
We use `sops-nix` to manage secrets, separating between Production and Staging.
|
||||
|
||||
**Prerequisites:** Install `age` ([age documentation](https://github.com/FiloSottile/age)).
|
||||
|
||||
1. **Generate the Production Master Key**
|
||||
|
||||
```bash
|
||||
age-keygen -o prod-master.age
|
||||
```
|
||||
|
||||
It is recommended to generate this key on a secure offline workstation. The private key should be stored in a secure offline location, such as a USB drive or printed on paper. Avoid storing this private key on any server.
|
||||
|
||||
2. **Generate the Staging Master Key**
|
||||
|
||||
```bash
|
||||
age-keygen -o staging-master.age
|
||||
```
|
||||
|
||||
3. **Update Configuration**
|
||||
|
||||
Replace both placeholders in `.sops.yaml` with the newly generated **public keys**. Commit and push this change.
|
||||
|
||||
## Proxmox Host Initialization & Authentication
|
||||
|
||||
The CI/CD pipeline needs restricted API access to Proxmox to provision Virtual Machines. We use OpenTofu to provision the bare-metal host.
|
||||
|
||||
### Find your Raw Disk ID
|
||||
|
||||
The current OpenTofu configuration assumes that your system has a 2TB disk attached to the Proxmox host. This disk will be formatted and used for VM storage.
|
||||
|
||||
1. Determine the persistent hardware ID of your 2TB disk, using the following command:
|
||||
Before bootstrapping your host, fork or clone this repository and make the following changes to match your environment:
|
||||
|
||||
### Hardware Identifiers
|
||||
- **Find your NVMe/Disk UUID**: Log into your fresh Proxmox host and run:
|
||||
```bash
|
||||
ls -l /dev/disk/by-id/
|
||||
```
|
||||
Identify your primary data disk (e.g. `nvme-eui...` or `wwn-0x...`).
|
||||
- **Update OpenTofu Config**: Open `opentofu/nodes/mikoshi/main.tf` (you may want to rename `mikoshi` to your host's name) and replace the `disk` ID inside the `zpool` resource with your hardware UUID.
|
||||
|
||||
It is recommended to use the persistent hardware ID (e.g., `wwn-0x...`, `nvme-eui...`, or `ata-...`) instead of `/dev/sdb` because the latter can change between reboots.
|
||||
### Identity & Access
|
||||
- **SSH Keys**: The GitOps Control Center needs an SSH key for disaster recovery.
|
||||
- Update the Cloud-Init SSH key in `opentofu/nodes/mikoshi/main.tf` under the `user_account` block.
|
||||
- Update the permanent NixOS SSH key in `nixos/users/admin/default.nix`.
|
||||
- **Secrets (SOPS)**: Replace the placeholder tokens in the Control Center host config (e.g., `nixos/hosts/izanagi/secrets.yaml` if you haven't renamed it) with your actual Proxmox API token and Forgejo token. Encrypt this file with your own `sops` Age key.
|
||||
|
||||
2. **Action Required:** Provide this ID to the system or update the `opentofu/nodes/pve-new/main.tf` configuration with this ID.
|
||||
### Hostname & Naming Schema
|
||||
If your Proxmox host or your Control Center has a different name:
|
||||
- Rename the folders in `opentofu/nodes/` and `nixos/hosts/`.
|
||||
- Update the `node_name` inside your OpenTofu `main.tf`.
|
||||
- Update `nixos/flake.nix` to reflect your new host names.
|
||||
|
||||
### Apply the OpenTofu Host State
|
||||
## Execute the Bootstrap
|
||||
|
||||
**Prerequisites:** Install `opentofu` ([OpenTofu installation](https://opentofu.org/docs/intro/install/)).
|
||||
Once you have pushed your adapted codebase to your Git server, SSH into your fresh Proxmox node as `root` and run the bootstrap script:
|
||||
|
||||
1. Initialize and apply the state. You will be prompted to enter your `root@pam` Proxmox credentials.
|
||||
```bash
|
||||
curl -fsSL https://git.your-server.com/your-repo/raw/branch/main/scripts/bootstrap.sh | bash
|
||||
```
|
||||
|
||||
```bash
|
||||
# Navigate to the OpenTofu host directory
|
||||
cd opentofu/host
|
||||
# Initialize and apply the OpenTofu configuration
|
||||
tofu init
|
||||
tofu apply
|
||||
```
|
||||
### What this script does automatically:
|
||||
- **Fixes APT Repositories**: Disables enterprise repositories and adds community repositories.
|
||||
- **Fixes NIC Offloading**: Installs a systemd service to safely disable TSO/GSO/GRO on physical interfaces to prevent network drops.
|
||||
- **Installs OpenTofu**: Pulls the official binaries.
|
||||
- **Applies Host State**: Runs `tofu apply` which:
|
||||
- Formats your specified disk into the `data` ZFS pool.
|
||||
- Sets laptop lid switch to ignore (if applicable).
|
||||
- Spins up the GitOps Control Center VM.
|
||||
|
||||
2. Upon successful completion, OpenTofu will output a secure **API Token**. Copy this token securely.
|
||||
## Post-Bootstrap
|
||||
|
||||
## Forgejo Secrets Configuration
|
||||
Once the bootstrap script completes, the Control Center VM will boot, initialize via Cloud-Init, and automatically start pulling this git repository.
|
||||
|
||||
The CI/CD actions require access to the Proxmox token and the staging secret key.
|
||||
|
||||
1. Navigate to your Forgejo Web UI.
|
||||
2. Go to **Settings > Actions > Secrets** for this repository.
|
||||
3. Add the following repository secrets:
|
||||
* `PROXMOX_TOKEN_SECRET`: Paste the token generated from Step 2.
|
||||
* `RENOVATE_TOKEN`: Create a Personal Access Token (PAT) for your user in Forgejo with read/write access to code and pull requests, and paste it here.
|
||||
|
||||
## Staging Golden Key Provisioning (Proxmox Snippet)
|
||||
|
||||
Instead of relying on Forgejo CI/CD to store the staging private key, we use a secure hypervisor-level Cloud-Init snippet.
|
||||
|
||||
1. SSH into your Proxmox server (`pve`).
|
||||
2. Create the Cloud-Init snippet file:
|
||||
|
||||
```bash
|
||||
cat << 'EOF' > /var/lib/vz/snippets/staging-key.yaml
|
||||
#cloud-config
|
||||
write_files:
|
||||
- path: /var/lib/sops-nix/key.txt
|
||||
permissions: '0600'
|
||||
content: |
|
||||
AGE-SECRET-KEY-1... (paste your staging-master private key here)
|
||||
runcmd:
|
||||
- echo "Staging age key injected successfully."
|
||||
EOF
|
||||
```
|
||||
|
||||
3. This completely removes the secret from Forgejo. When OpenTofu spins up a staging VM, it simply tells Proxmox to attach this local snippet!
|
||||
|
||||
## TrueNAS API Security (RBAC)
|
||||
|
||||
To prevent the CI/CD pipeline from having `root` access to your TrueNAS server, you must run the RBAC bootstrap script to create a restricted user (`forgejo-ci`) that can *only* clone datasets for staging, not destroy production data.
|
||||
|
||||
1. Ensure you have network access to your TrueNAS host.
|
||||
2. Execute the RBAC setup script:
|
||||
|
||||
```bash
|
||||
./scripts/truenas-rbac-setup.sh
|
||||
```
|
||||
|
||||
3. Provide your TrueNAS IP and the `root` Admin API Token when prompted.
|
||||
4. The script will automatically create the custom `ci-runner-role` and the `forgejo-ci` user.
|
||||
5. Follow the terminal output instructions to log into the TrueNAS Web UI as the new user and generate the restricted API token.
|
||||
6. Use this restricted token for the `TRUENAS_API_KEY` secret in Forgejo.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Once these bootstrap steps are complete, the foundational authentication is in place. The Forgejo CI actions will now have the necessary permissions to build images, provision VMs, and test staging environments autonomously and securely.
|
||||
From this point on, **you no longer need to log into the Proxmox host.** All future changes to VMs, networks, or applications should be done declaratively via Pull Requests to your repository!
|
||||
|
|
|
|||
|
|
@ -70,7 +70,7 @@
|
|||
};
|
||||
|
||||
hosts = {
|
||||
# Hosts will be populated here as they are migrated to the v2 branch.
|
||||
izanagi.modules = [ ./hosts/izanagi ];
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
|
|||
52
nixos/hosts/izanagi/default.nix
Normal file
52
nixos/hosts/izanagi/default.nix
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
{ config, pkgs, ... }:
|
||||
|
||||
{
|
||||
networking.hostName = "izanagi";
|
||||
|
||||
homelab = {
|
||||
# Enable the standard guest VM configuration
|
||||
virtualisation.guest.enable = true;
|
||||
|
||||
# Enable standard GitOps for the Control Center itself
|
||||
common.gitops.enable = true;
|
||||
# We might pull from our Forgejo instance eventually, but for bootstrap
|
||||
# it might need to pull from Github or the local Gitea if it's up.
|
||||
# common.gitops.repoUrl = "https://git.depeuter.dev/Bos55/nix-config.git";
|
||||
|
||||
services = {
|
||||
openssh.enable = true;
|
||||
|
||||
# Enable the Hypervisor GitOps service to provision OTHER VMs
|
||||
hypervisor-gitops = {
|
||||
enable = true;
|
||||
repoUrl = "https://git.depeuter.dev/Bos55/nix-config.git";
|
||||
pollInterval = "hourly";
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
sops = {
|
||||
defaultSopsFile = ./secrets.yaml;
|
||||
defaultSopsFormat = "yaml";
|
||||
age.keyFile = "/var/lib/sops-age/keys.txt"; # Injected via Cloud-Init during Phase 4
|
||||
|
||||
secrets = {
|
||||
proxmox_api_token.owner = "root";
|
||||
forgejo_token.owner = "root";
|
||||
};
|
||||
|
||||
templates."hypervisor-gitops.env".content = ''
|
||||
PROXMOX_VE_API_TOKEN=${config.sops.placeholder.proxmox_api_token}
|
||||
FORGEJO_TOKEN=${config.sops.placeholder.forgejo_token}
|
||||
PROXMOX_VE_ENDPOINT=https://mikoshi:8006/
|
||||
PROXMOX_VE_INSECURE=true
|
||||
'';
|
||||
};
|
||||
|
||||
# Make the secrets available to the hypervisor-gitops service via EnvironmentFile
|
||||
systemd.services.hypervisor-gitops.serviceConfig.EnvironmentFile = [
|
||||
config.sops.templates."hypervisor-gitops.env".path
|
||||
];
|
||||
|
||||
system.stateVersion = "24.05";
|
||||
}
|
||||
5
nixos/hosts/izanagi/secrets.yaml
Normal file
5
nixos/hosts/izanagi/secrets.yaml
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
# This is a placeholder SOPS file.
|
||||
# You must encrypt it with sops using your age key before deploying.
|
||||
# sops -e -i secrets.yaml
|
||||
proxmox_api_token: "PLACEHOLDER_TOKEN"
|
||||
forgejo_token: "PLACEHOLDER_TOKEN"
|
||||
|
|
@ -6,42 +6,7 @@ let
|
|||
updateScript = pkgs.writeShellApplication {
|
||||
name = "homelab-gitops-update";
|
||||
runtimeInputs = [ pkgs.git pkgs.nixos-rebuild pkgs.jq pkgs.coreutils ];
|
||||
text = ''
|
||||
set -euo pipefail
|
||||
|
||||
REMOTE_URL="${cfg.repoUrl}"
|
||||
BRANCH="${cfg.branch}"
|
||||
|
||||
echo "Checking remote hash for $REMOTE_URL branch $BRANCH..."
|
||||
|
||||
# Fetch remote hash, fallback to unknown if it fails
|
||||
REMOTE_HASH=$(git ls-remote "$REMOTE_URL" "refs/heads/$BRANCH" | awk '{print $1}' || true)
|
||||
|
||||
if [ -z "$REMOTE_HASH" ]; then
|
||||
echo "WARNING: Could not fetch remote hash. Forcing rebuild to be safe."
|
||||
REMOTE_HASH="unknown_remote"
|
||||
fi
|
||||
|
||||
LOCAL_HASH="unknown_local"
|
||||
if [ -f /run/current-system/configurationRevision ]; then
|
||||
LOCAL_HASH=$(cat /run/current-system/configurationRevision)
|
||||
fi
|
||||
|
||||
echo "Remote hash: $REMOTE_HASH"
|
||||
echo "Local hash: $LOCAL_HASH"
|
||||
|
||||
if [ "$REMOTE_HASH" = "$LOCAL_HASH" ] && [ "$REMOTE_HASH" != "unknown_remote" ] && [ "$LOCAL_HASH" != "unknown" ]; then
|
||||
echo "Hashes match. No update needed."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Hashes differ or unknown. Triggering nixos-rebuild..."
|
||||
|
||||
# Trigger the build and switch
|
||||
nixos-rebuild switch --flake "git+$REMOTE_URL?dir=nixos&ref=$BRANCH"
|
||||
|
||||
echo "Update successful."
|
||||
'';
|
||||
text = builtins.readFile ../../../../scripts/nixos-sync.sh;
|
||||
};
|
||||
|
||||
in {
|
||||
|
|
@ -68,14 +33,13 @@ in {
|
|||
};
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
# 1. Systemd Service and Timer for polling
|
||||
systemd.services.homelab-gitops = {
|
||||
description = "Homelab GitOps Update Service";
|
||||
wants = [ "network-online.target" ];
|
||||
after = [ "network-online.target" ];
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
ExecStart = "${updateScript}/bin/homelab-gitops-update";
|
||||
ExecStart = "${updateScript}/bin/homelab-gitops-update ${cfg.repoUrl} ${cfg.branch}";
|
||||
# Must run as root to rebuild the system
|
||||
User = "root";
|
||||
};
|
||||
|
|
@ -91,7 +55,6 @@ in {
|
|||
};
|
||||
};
|
||||
|
||||
# 2. Webhook listener for instant trigger
|
||||
sops.secrets."webhook-secret" = {};
|
||||
|
||||
services.webhook = {
|
||||
|
|
@ -121,7 +84,6 @@ in {
|
|||
# Inject the secret as an environment variable into the webhook service
|
||||
systemd.services.webhook.serviceConfig.EnvironmentFile = config.sops.secrets."webhook-secret".path;
|
||||
|
||||
# 3. Builder Configuration
|
||||
sops.secrets."builder-ssh-key" = lib.mkIf cfg.useBuilder {};
|
||||
|
||||
nix.buildMachines = lib.mkIf cfg.useBuilder [
|
||||
|
|
|
|||
|
|
@ -2,5 +2,6 @@
|
|||
imports = [
|
||||
./actions
|
||||
./openssh
|
||||
./hypervisor-gitops
|
||||
];
|
||||
}
|
||||
|
|
|
|||
64
nixos/modules/services/hypervisor-gitops/default.nix
Normal file
64
nixos/modules/services/hypervisor-gitops/default.nix
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
{ config, lib, pkgs, ... }:
|
||||
|
||||
with lib;
|
||||
|
||||
let
|
||||
cfg = config.homelab.services.hypervisor-gitops;
|
||||
|
||||
hypervisorSyncScript = pkgs.writeShellApplication {
|
||||
name = "hypervisor-sync";
|
||||
runtimeInputs = with pkgs; [ git opentofu coreutils ];
|
||||
text = builtins.readFile ../../../../scripts/hypervisor-sync.sh;
|
||||
};
|
||||
in {
|
||||
options.homelab.services.hypervisor-gitops = {
|
||||
enable = mkEnableOption "Hypervisor GitOps Service";
|
||||
|
||||
repoUrl = mkOption {
|
||||
type = types.str;
|
||||
description = "The URL of the git repository to pull";
|
||||
};
|
||||
|
||||
# TODO Replace with webhooks
|
||||
pollInterval = mkOption {
|
||||
type = types.str;
|
||||
default = "hourly";
|
||||
description = "Systemd calendar event for polling interval";
|
||||
};
|
||||
};
|
||||
|
||||
config = mkIf cfg.enable {
|
||||
environment.systemPackages = with pkgs; [
|
||||
git
|
||||
opentofu
|
||||
];
|
||||
|
||||
systemd.services.hypervisor-gitops = {
|
||||
description = "Hypervisor GitOps Polling Service";
|
||||
|
||||
# We need network access to reach Forgejo and Proxmox API
|
||||
after = [ "network-online.target" ];
|
||||
wants = [ "network-online.target" ];
|
||||
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
User = "root"; # Needs root to read SOPS secrets potentially
|
||||
|
||||
# We will create a state directory for the repo
|
||||
StateDirectory = "hypervisor-gitops";
|
||||
WorkingDirectory = "/var/lib/hypervisor-gitops";
|
||||
|
||||
ExecStart = "${hypervisorSyncScript}/bin/hypervisor-sync ${cfg.repoUrl} opentofu/nodes/mikoshi";
|
||||
};
|
||||
};
|
||||
|
||||
systemd.timers.hypervisor-gitops = {
|
||||
description = "Timer for Hypervisor GitOps Service";
|
||||
wantedBy = [ "timers.target" ];
|
||||
timerConfig = {
|
||||
OnCalendar = cfg.pollInterval;
|
||||
Persistent = true;
|
||||
};
|
||||
};
|
||||
};
|
||||
}
|
||||
|
|
@ -9,7 +9,7 @@ in {
|
|||
type = lib.types.listOf lib.types.str;
|
||||
default = [
|
||||
# HomeLab > NixOS > admin > ssh
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGWIOOEqTy8cWKpENVbzD4p7bsQgQb/Dgpzk8i0dZ00T"
|
||||
"sk-ssh-ed25519@openssh.com AAAAGnNrLXNzaC1lZDI1NTE5QG9wZW5zc2guY29tAAAAIHOoTp+e6qWGn4Sco5CZ6G0zrX5NAQBpLlDVirncJ/HqAAAABHNzaDo="
|
||||
];
|
||||
};
|
||||
};
|
||||
|
|
|
|||
123
opentofu/modules/proxmox-node/main.tf
Normal file
123
opentofu/modules/proxmox-node/main.tf
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
terraform {
|
||||
required_providers {
|
||||
proxmox = {
|
||||
source = "bpg/proxmox"
|
||||
version = "~> 0.61.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Bare Metal Host Configurations (SSH Provisioning)
|
||||
|
||||
# We use a null_resource to run imperative commands on the Debian host
|
||||
# that are not currently supported by the bpg/proxmox provider.
|
||||
resource "null_resource" "bare_metal_setup" {
|
||||
triggers = {
|
||||
node = var.node_name
|
||||
data_disk_id = var.data_disk_id
|
||||
pool_name = var.zfs_pool_name
|
||||
}
|
||||
|
||||
connection {
|
||||
type = "ssh"
|
||||
user = "root"
|
||||
# Assuming running locally on the node during bootstrap, or via SSH if run from a laptop.
|
||||
# We default to local host if run from Control Center, but for flexibility we use the endpoint.
|
||||
host = var.node_name
|
||||
agent = true
|
||||
}
|
||||
|
||||
provisioner "remote-exec" {
|
||||
inline = [
|
||||
# Set laptop lid switch to ignore (prevents sleeping when closed)
|
||||
"sed -i 's/^#\\?HandleLidSwitch=.*/HandleLidSwitch=ignore/' /etc/systemd/logind.conf",
|
||||
"systemctl restart systemd-logind",
|
||||
|
||||
# Format the ZFS pool if it doesn't already exist
|
||||
"zpool list ${var.zfs_pool_name} || zpool create -f ${var.zfs_pool_name} /dev/disk/by-id/${var.data_disk_id}",
|
||||
|
||||
# Register the ZFS pool in Proxmox if it's not already registered
|
||||
"pvesm status -storage ${var.zfs_pool_name} || pvesm add zfspool ${var.zfs_pool_name} --pool ${var.zfs_pool_name} --content images,rootdir"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
# Resource Pools
|
||||
|
||||
# Creates logical groups for organizing and securing VMs.
|
||||
resource "proxmox_virtual_environment_pool" "core" {
|
||||
pool_id = "core"
|
||||
comment = "Critical, public-facing services (reverse proxy, DNS)"
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_pool" "production" {
|
||||
pool_id = "production"
|
||||
comment = "Stable services with backup strategies"
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_pool" "dev" {
|
||||
pool_id = "dev"
|
||||
comment = "Persistent but non-production services, run 24/7"
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_pool" "staging" {
|
||||
pool_id = "staging"
|
||||
comment = "Ephemeral test VMs managed by Pull-Based PR webhooks"
|
||||
}
|
||||
|
||||
# API Tokens for Control Center
|
||||
|
||||
# Creates a restricted user for the Control Center VM to manage the cluster.
|
||||
resource "proxmox_virtual_environment_role" "control_center_role" {
|
||||
role_id = "ControlCenter"
|
||||
|
||||
privileges = [
|
||||
"VM.Allocate",
|
||||
"VM.Audit",
|
||||
"VM.Clone",
|
||||
"VM.Config.CDROM",
|
||||
"VM.Config.CPU",
|
||||
"VM.Config.Disk",
|
||||
"VM.Config.HWType",
|
||||
"VM.Config.Memory",
|
||||
"VM.Config.Network",
|
||||
"VM.Config.Options",
|
||||
"VM.Monitor",
|
||||
"VM.PowerMgmt",
|
||||
"Datastore.AllocateSpace",
|
||||
"Datastore.Audit",
|
||||
"SDN.Use"
|
||||
]
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_user" "control_center_user" {
|
||||
user_id = "control-center@pve"
|
||||
comment = "Managed by OpenTofu - Used by Control Center for GitOps"
|
||||
}
|
||||
|
||||
# Grant full VM access to the control center
|
||||
resource "proxmox_virtual_environment_acl" "control_center_vms" {
|
||||
user_id = proxmox_virtual_environment_user.control_center_user.user_id
|
||||
role_id = proxmox_virtual_environment_role.control_center_role.role_id
|
||||
path = "/vms"
|
||||
}
|
||||
|
||||
# Grant datastore access
|
||||
resource "proxmox_virtual_environment_acl" "control_center_storage" {
|
||||
user_id = proxmox_virtual_environment_user.control_center_user.user_id
|
||||
role_id = proxmox_virtual_environment_role.control_center_role.role_id
|
||||
path = "/storage/${var.zfs_pool_name}"
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_user_token" "control_center_token" {
|
||||
user_id = proxmox_virtual_environment_user.control_center_user.user_id
|
||||
token_id = "gitops"
|
||||
privsep = false
|
||||
comment = "Stored only on the Control Center VM"
|
||||
}
|
||||
|
||||
output "control_center_api_token" {
|
||||
value = proxmox_virtual_environment_user_token.control_center_token.value
|
||||
sensitive = true
|
||||
description = "The secret API token for control-center@pve. This will be injected into the Control Center SOPS config."
|
||||
}
|
||||
15
opentofu/modules/proxmox-node/variables.tf
Normal file
15
opentofu/modules/proxmox-node/variables.tf
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
variable "node_name" {
|
||||
type = string
|
||||
description = "The name of the Proxmox node (e.g. pve)"
|
||||
}
|
||||
|
||||
variable "data_disk_id" {
|
||||
type = string
|
||||
description = "The persistent block device ID for the data disk (e.g. wwn-0x500...)"
|
||||
}
|
||||
|
||||
variable "zfs_pool_name" {
|
||||
type = string
|
||||
default = "data"
|
||||
description = "The name of the ZFS pool to create on the data disk"
|
||||
}
|
||||
84
opentofu/nodes/mikoshi/main.tf
Normal file
84
opentofu/nodes/mikoshi/main.tf
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
terraform {
|
||||
required_providers {
|
||||
proxmox = {
|
||||
source = "bpg/proxmox"
|
||||
version = "~> 0.61.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "proxmox" {
|
||||
# Endpoint and credentials will be picked up from environment variables
|
||||
# or passed via the bootstrap script.
|
||||
# PROXMOX_VE_ENDPOINT
|
||||
# PROXMOX_VE_USERNAME
|
||||
# PROXMOX_VE_PASSWORD
|
||||
# PROXMOX_VE_INSECURE=true
|
||||
}
|
||||
|
||||
module "proxmox_node" {
|
||||
source = "../../modules/proxmox-node"
|
||||
node_name = "mikoshi"
|
||||
data_disk_id = "nvme-KXG80ZNV2T04_NVMe_KIOXIA_2048GB_241C11Y5EHAK"
|
||||
zfs_pool_name = "data"
|
||||
}
|
||||
|
||||
output "control_center_api_token" {
|
||||
value = module.proxmox_node.control_center_api_token
|
||||
sensitive = true
|
||||
description = "The secret API token for control-center@pve."
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_vm" "control_center" {
|
||||
depends_on = [module.proxmox_node]
|
||||
|
||||
name = "izanagi"
|
||||
description = "Managed by OpenTofu - GitOps Control Center"
|
||||
tags = ["infrastructure", "gitops"]
|
||||
node_name = "mikoshi"
|
||||
vm_id = 100001000
|
||||
|
||||
on_boot = true
|
||||
|
||||
pool_id = "core"
|
||||
|
||||
cpu {
|
||||
cores = 2
|
||||
type = "x86-64-v2-AES"
|
||||
}
|
||||
|
||||
memory {
|
||||
dedicated = 2048
|
||||
}
|
||||
|
||||
agent {
|
||||
enabled = true
|
||||
}
|
||||
|
||||
network_device {
|
||||
bridge = "vmbr0"
|
||||
}
|
||||
|
||||
disk {
|
||||
datastore_id = "data"
|
||||
file_id = "local:iso/nixos-minimal.iso" # TODO Replace with actual ISO or use Clone
|
||||
interface = "scsi0"
|
||||
size = 20
|
||||
file_format = "raw"
|
||||
}
|
||||
|
||||
# Cloud-Init for initial SSH access and SOPS age key injection
|
||||
initialization {
|
||||
ip_config {
|
||||
ipv4 {
|
||||
address = "dhcp"
|
||||
}
|
||||
}
|
||||
user_account {
|
||||
username = "gh0st"
|
||||
keys = [
|
||||
"sk-ssh-ed25519@openssh.com AAAAGnNrLXNzaC1lZDI1NTE5QG9wZW5zc2guY29tAAAAIHOoTp+e6qWGn4Sco5CZ6G0zrX5NAQBpLlDVirncJ/HqAAAABHNzaDo="
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,71 +0,0 @@
|
|||
terraform {
|
||||
required_providers {
|
||||
proxmox = {
|
||||
source = "bpg/proxmox"
|
||||
version = "~> 0.61.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# This bootstrap state must be run manually ONCE with the root@pam credentials
|
||||
# to establish the restricted terraform@pve user for the rest of the CI pipeline.
|
||||
provider "proxmox" {
|
||||
# Configuration can be passed via environment variables:
|
||||
# PROXMOX_VE_ENDPOINT=https://your-proxmox-ip:8006/
|
||||
# PROXMOX_VE_USERNAME=root@pam
|
||||
# PROXMOX_VE_PASSWORD=your-root-password
|
||||
# PROXMOX_VE_INSECURE=true # If using self-signed certs
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_role" "terraform_prov" {
|
||||
role_id = "TerraformProv"
|
||||
|
||||
privileges = [
|
||||
"VM.Allocate",
|
||||
"VM.Audit",
|
||||
"VM.Clone",
|
||||
"VM.Config.CDROM",
|
||||
"VM.Config.CPU",
|
||||
"VM.Config.Disk",
|
||||
"VM.Config.HWType",
|
||||
"VM.Config.Memory",
|
||||
"VM.Config.Network",
|
||||
"VM.Config.Options",
|
||||
"VM.Monitor",
|
||||
"VM.PowerMgmt",
|
||||
"Datastore.AllocateSpace",
|
||||
"Datastore.Audit",
|
||||
"SDN.Use"
|
||||
]
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_user" "terraform_user" {
|
||||
user_id = "terraform@pve"
|
||||
comment = "Managed by Terraform (proxmox-bootstrap) for GitOps CI/CD"
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_acl" "terraform_vms" {
|
||||
user_id = proxmox_virtual_environment_user.terraform_user.user_id
|
||||
role_id = proxmox_virtual_environment_role.terraform_prov.role_id
|
||||
path = "/vms"
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_acl" "terraform_storage" {
|
||||
user_id = proxmox_virtual_environment_user.terraform_user.user_id
|
||||
role_id = proxmox_virtual_environment_role.terraform_prov.role_id
|
||||
# Update this path to match your actual local-zfs or TrueNAS mounted storage
|
||||
path = "/storage/local-zfs"
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_user_token" "terraform_token" {
|
||||
user_id = proxmox_virtual_environment_user.terraform_user.user_id
|
||||
token_id = "tf-automation"
|
||||
privsep = false
|
||||
comment = "Token for Forgejo CI/CD to provision VMs"
|
||||
}
|
||||
|
||||
output "terraform_api_token" {
|
||||
value = proxmox_virtual_environment_user_token.terraform_token.value
|
||||
sensitive = true
|
||||
description = "The secret API token for terraform@pve. Save this to Forgejo Secrets as PROXMOX_VE_API_TOKEN."
|
||||
}
|
||||
76
scripts/bootstrap.sh
Normal file
76
scripts/bootstrap.sh
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
BANNER="==========================================================="
|
||||
printf "%s\n Proxmox Bootstrap\n%s\n\n" "$BANNER" "$BANNER"
|
||||
|
||||
echo "Applying post-pve-install fixes (fixing repos)..."
|
||||
# Remove enterprise repos and add non-subscription repos safely
|
||||
rm -f /etc/apt/sources.list.d/pve-enterprise.list
|
||||
|
||||
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list
|
||||
|
||||
# Disable the "No Valid Subscription" nag screen
|
||||
sed -i.bak "s/data.status !== 'Active'/false/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
|
||||
|
||||
echo ""
|
||||
echo "Applying NIC offloading fixes..."
|
||||
# The community script disables TSO, GSO, and GRO on physical interfaces.
|
||||
# We create a systemd service to ensure this applies on boot.
|
||||
cat <<'EOF' > /etc/systemd/system/nic-offload-fix.service
|
||||
[Unit]
|
||||
Description=Disable NIC offloading (TSO/GRO/GSO) for physical interfaces
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
# Iterate over all physical interfaces (excluding lo, bridges, veth, etc.)
|
||||
ExecStart=/bin/bash -c 'for dev in /sys/class/net/*; do if [ "$(basename "$dev")" != "lo" ] && [[ ! "$(basename "$dev")" =~ ^(vmbr|veth|fwbr|tap|bonding) ]]; then /usr/sbin/ethtool -K "$(basename "$dev")" tso off gso off gro off || true; fi; done'
|
||||
RemainAfterExit=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
systemctl enable --now nic-offload-fix.service || true
|
||||
|
||||
echo ""
|
||||
echo "Updating system and installing OpenTofu..."
|
||||
apt-get update
|
||||
# Install curl, git, gnupg, ethtool, and required apt dependencies
|
||||
apt-get install -y apt-transport-https ca-certificates curl git gnupg ethtool
|
||||
|
||||
# Install OpenTofu repository and binary
|
||||
install -m 0755 -d /etc/apt/keyrings
|
||||
curl -fsSL https://get.opentofu.org/opentofu.gpg | tee /etc/apt/keyrings/opentofu.gpg >/dev/null
|
||||
curl -fsSL https://packages.opentofu.org/opentofu/tofu/gpgkey | gpg --no-tty --batch --dearmor -o /etc/apt/keyrings/opentofu-repo.gpg >/dev/null
|
||||
chmod a+r /etc/apt/keyrings/opentofu.gpg /etc/apt/keyrings/opentofu-repo.gpg
|
||||
|
||||
printf "deb [signed-by=/etc/apt/keyrings/opentofu.gpg,/etc/apt/keyrings/opentofu-repo.gpg] https://packages.opentofu.org/opentofu/tofu/any/ any main\ndeb-src [signed-by=/etc/apt/keyrings/opentofu.gpg,/etc/apt/keyrings/opentofu-repo.gpg] https://packages.opentofu.org/opentofu/tofu/any/ any main\n" > /etc/apt/sources.list.d/opentofu.list
|
||||
chmod a+r /etc/apt/sources.list.d/opentofu.list
|
||||
|
||||
apt-get update
|
||||
apt-get install -y tofu
|
||||
|
||||
echo ""
|
||||
echo "Cloning the infrastructure repository..."
|
||||
cd /root
|
||||
if [ -d "nix-config" ]; then
|
||||
echo "Repository already exists. Pulling latest..."
|
||||
cd nix-config
|
||||
git pull
|
||||
else
|
||||
git clone https://git.depeuter.dev/Bos55/nix-config.git
|
||||
cd nix-config
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Bootstrapping Proxmox Host State..."
|
||||
cd opentofu/nodes/mikoshi
|
||||
|
||||
echo "Initializing OpenTofu..."
|
||||
tofu init -upgrade
|
||||
|
||||
echo "Applying bare-metal state..."
|
||||
tofu apply -auto-approve
|
||||
|
||||
printf "\n%s\n Bootstrap Complete!\n The ZFS pool, Resource Pools, and the Control Center\n VM have been provisioned.\n\n The Control Center is booting now. Once online, it will automatically\n pull this repository and provision the rest of your VMs!\n%s\n" "$BANNER" "$BANNER"
|
||||
39
scripts/hypervisor-sync.sh
Normal file
39
scripts/hypervisor-sync.sh
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
# This script pulls the latest changes from the Git repository
|
||||
# and runs OpenTofu to provision the hypervisor state.
|
||||
# Usage: ./hypervisor-sync.sh <REPO_URL> <OPENTOFU_DIR>
|
||||
|
||||
REPO_URL=${1:-"https://git.depeuter.dev/Bos55/nix-config.git"}
|
||||
OPENTOFU_DIR=${2:-"opentofu/nodes/mikoshi"}
|
||||
|
||||
echo "Starting Hypervisor GitOps sync..."
|
||||
|
||||
if [ ! -d "nix-config" ]; then
|
||||
echo "Cloning repository from $REPO_URL..."
|
||||
git clone "$REPO_URL" nix-config
|
||||
fi
|
||||
|
||||
cd nix-config || exit
|
||||
|
||||
git fetch origin main
|
||||
|
||||
LOCAL=$(git rev-parse HEAD)
|
||||
REMOTE=$(git rev-parse origin/main)
|
||||
|
||||
if [ "$LOCAL" = "$REMOTE" ]; then
|
||||
echo "Already up to date. Nothing to do."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Changes detected. Updating from $LOCAL to $REMOTE..."
|
||||
git reset --hard origin/main
|
||||
|
||||
echo "Applying OpenTofu changes in $OPENTOFU_DIR..."
|
||||
cd "$OPENTOFU_DIR" || exit
|
||||
|
||||
tofu init -upgrade
|
||||
|
||||
tofu apply -auto-approve
|
||||
|
||||
echo "Hypervisor GitOps sync completed successfully."
|
||||
40
scripts/nixos-sync.sh
Normal file
40
scripts/nixos-sync.sh
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
# This script checks the remote Git repository for changes
|
||||
# and triggers a nixos-rebuild if a new commit is found.
|
||||
# Usage: ./nixos-sync.sh <REPO_URL> <BRANCH>
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_URL=${1:-"https://git.depeuter.dev/Bos55/nix-config.git"}
|
||||
BRANCH=${2:-"main"}
|
||||
|
||||
echo "Checking remote hash for $REPO_URL branch $BRANCH..."
|
||||
|
||||
# Fetch remote hash, fallback to unknown if it fails
|
||||
REMOTE_HASH=$(git ls-remote "$REPO_URL" "refs/heads/$BRANCH" | awk '{print $1}' || true)
|
||||
|
||||
if [ -z "$REMOTE_HASH" ]; then
|
||||
echo "WARNING: Could not fetch remote hash. Forcing rebuild to be safe."
|
||||
REMOTE_HASH="unknown_remote"
|
||||
fi
|
||||
|
||||
LOCAL_HASH="unknown_local"
|
||||
if [ -f /run/current-system/configurationRevision ]; then
|
||||
LOCAL_HASH=$(cat /run/current-system/configurationRevision)
|
||||
fi
|
||||
|
||||
echo "Remote hash: $REMOTE_HASH"
|
||||
echo "Local hash: $LOCAL_HASH"
|
||||
|
||||
if [ "$REMOTE_HASH" = "$LOCAL_HASH" ] && [ "$REMOTE_HASH" != "unknown_remote" ] && [ "$LOCAL_HASH" != "unknown" ]; then
|
||||
echo "Hashes match. No update needed."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Hashes differ or unknown. Triggering nixos-rebuild..."
|
||||
|
||||
# Trigger the build and switch
|
||||
nixos-rebuild switch --flake "git+$REPO_URL?dir=nixos&ref=$BRANCH"
|
||||
|
||||
echo "Update successful."
|
||||
Loading…
Add table
Add a link
Reference in a new issue