From 953e10248364f566c805802a0771ddaaf7170176 Mon Sep 17 00:00:00 2001 From: Tibo De Peuter Date: Sat, 4 Apr 2026 18:12:33 +0200 Subject: [PATCH] docs(hpc): add docs/HPC.md and simplify README - docs/HPC.md: cluster table, setup with vsc-venv, interactive debug (shell + JupyterLab on donphan), PBS batch submission, dependency mgmt - README.md: HPC section condensed to a single link to docs/HPC.md --- README.md | 24 +--------- docs/HPC.md | 129 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+), 22 deletions(-) create mode 100644 docs/HPC.md diff --git a/README.md b/README.md index dfe3d4d..085f3d5 100644 --- a/README.md +++ b/README.md @@ -16,26 +16,6 @@ example command: uv run src/train.py --model_name my_model --epochs 50 --batch_size 32 ``` -## HPC Integration +## HPC -To run training on the VSC HPC (Ghent Tier 2): - -### Setup - -Run the installation script once on a login node. This sets up a virtual environment in `$VSC_DATA` with the required system modules (JAX, Flax, WandB) and lightweight dependencies. - -```bash -bash scripts/hpc_install.sh -``` - -### Smoke Test -Verify everything runs on a compute node: -```bash -python src/train.py --config configs/hpc_smoke_test.yaml -``` - -### Submitting Jobs -Submit long-running experiments via the provided Slurm script: -```bash -sbatch scripts/hpc_train.slurm -``` +See **[docs/HPC.md](docs/HPC.md)** for the full guide, including environment setup, cluster selection, interactive debugging, and job submission. diff --git a/docs/HPC.md b/docs/HPC.md new file mode 100644 index 0000000..89d49bf --- /dev/null +++ b/docs/HPC.md @@ -0,0 +1,129 @@ +# HPC Guide (Ghent University Tier-2) + +Full documentation: + +## Cluster Selection + +Choose the appropriate cluster before submitting a job with `module swap cluster/`. The default login cluster is **doduo**. + +| Cluster | Type | Use case | +|-----------|-------------------------|-----------------------------------------------| +| `donphan` | Interactive / debug GPU | First-time setup, interactive debugging | +| `doduo` | CPU (default login) | Rapid iteration, CPU-only smoke tests | +| `joltik` | GPU (A100 ¼-slice) | Standard training runs | +| `accelgor`| GPU (A100 full) | Large-scale / long experiments | +| `litleo` | GPU | Alternative GPU option | + +> **Rule:** use at most **1 GPU per group at a time** on shared GPU clusters. +> Check the current queue load at . + +--- + +## Initial Environment Setup + +Run **once** from a login node after cloning the repository: + +```bash +# 1. Connect via the web portal → HPC Login → Interactive Apps > Shell (tmux) +# Set cluster to "donphan (interactive/debug)" + +# 2. Clone the project (if not done yet) +git clone +cd 2026SEL3-project-BrittleStar + +# 3. Run the install script +bash scripts/hpc/install.sh +``` + +This uses the official [`vsc-venv`](https://docs.hpc.ugent.be/Linux/setting_up_python_virtual_environments/#vsc-venv-python-virtual-environment-wrapper-script) wrapper to: +- Load the EasyBuild modules listed in `env/hpc/modules.txt` (JAX, Flax, WandB, …) +- Create a per-cluster virtual environment in `$VSC_DATA` +- Pip-install the remaining packages from `env/hpc/requirements.txt` +- Register a Jupyter kernel named `sel3_` + +> **Note:** virtual environments are cluster-specific. Re-run the script if you switch to a new cluster. + +--- + +## Interactive Debugging on donphan + +The `donphan` cluster provides quick access to a fractional A100 GPU and is ideal for verifying your environment end-to-end before submitting batch jobs. + +### Option A — Interactive shell session + +```bash +# Swap to the debug cluster (from the login node) +module swap cluster/donphan + +# Request an interactive job (1 node, 4 cores) +qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00 + +# Once inside the job, activate your environment +cd "$PBS_O_WORKDIR" # or cd to your project root if not set +module load vsc-venv +source vsc-venv --activate --modules env/hpc/modules.txt + +# Run the smoke test +export MUJOCO_GL=egl +python src/train.py --config configs/hpc_smoke_test.yaml +``` + +### Option B — JupyterLab session (HPC web portal) + +1. In the web portal go to **Interactive Apps → JupyterLab RHEL9** +2. Set the following options: + + | Option | Value | + |---------------------|---------------------------------| + | Cluster | `donphan (interactive/debug)` | + | Number of nodes | 1 | + | Number of cores | 4 | + | JupyterLab version | `4.2.5 GCCcore-13.3.0` | + | Custom code | *(leave blank — vsc-venv handles modules)* | + +3. Click **Launch**, wait for the session to start, then **Connect**. +4. In JupyterLab, select the kernel **`SEL3 ()`**. +5. Verify GPU access: + + ```python + import jax + print(jax.default_backend()) # expected: 'gpu' + print(jax.devices()) # expected: [CudaDevice(id=0)] + ``` + +> **Warning:** JAX can only be loaded by one kernel at a time. Shut down other kernels before switching notebooks. + +--- + +## Submitting Batch Training Jobs + +```bash +# Default cluster (joltik — one A100 GPU slice) +qsub scripts/hpc/train.pbs + +# Switch to a different GPU cluster first +module swap cluster/accelgor +qsub scripts/hpc/train.pbs +``` + +Monitor your jobs: + +```bash +qstat # list your jobs +qstat -f # detailed info for a specific job +qdel # cancel a job +``` + +Logs are written to `runs/pbs_.out` and `runs/pbs_.err`. + +--- + +## Managing Dependencies + +`env/hpc/requirements.txt` is the **source of truth** for pip packages on the HPC. It is automatically regenerated by CI whenever `pyproject.toml` changes. To regenerate locally: + +```bash +uv run scripts/export_hpc_requirements.py +``` + +Do **not** edit `env/hpc/requirements.txt` by hand — edit `pyproject.toml` instead.