diff --git a/docs/HPC.md b/docs/HPC.md index 5c7b02c..040d0ac 100644 --- a/docs/HPC.md +++ b/docs/HPC.md @@ -6,17 +6,49 @@ Full documentation: Choose the appropriate cluster before submitting a job with `module swap cluster/`. The default login cluster is **doduo**. +| Cluster | Type | Use case | +|-----------|-------------------------|-----------------------------------------------| +| `donphan` | Interactive / debug GPU | First-time setup, interactive debugging | +| `doduo` | CPU (default login) | Rapid iteration, CPU-only smoke tests | +| `joltik` | GPU (A100 ¼-slice) | Standard training runs | +| `accelgor`| GPU (A100 full) | Large-scale / long experiments | +| `litleo` | GPU | Alternative GPU option | + +> **Rule:** use at most **1 GPU per group at a time** on shared GPU clusters. > Check the current queue load at . +--- + +## Storage Overview + +The HPC provides three filesystems for different purposes. Understanding this is critical to avoid filling up your home directory. + +| Variable | Typical size | Purpose | +|-----------------|-------------|------------------------------------------------------| +| `$VSC_HOME` | ~3 GB | Config files, SSH keys, project source code only | +| `$VSC_DATA` | ~25 GB | Persistent outputs: trained models, final logs | +| `$VSC_SCRATCH` | Large | Fast I/O during jobs: caches, intermediate files | + +**Important:** +- Clone the repository into `$VSC_HOME` — it is small in size and accessible from all clusters. +- All caches (pip, uv, matplotlib) must be redirected to `$VSC_SCRATCH` to avoid filling `$VSC_HOME`. +- Run outputs are written to `$VSC_SCRATCH` during the job (fast I/O) and copied to `$VSC_DATA` at the end for persistence. +- `$VSC_SCRATCH` may be purged periodically — do not use it as long-term storage. + +Check your quota: (Usage section). + +--- + ## Initial Environment Setup -Run **once** from a login node after cloning the repository: +Run **once** from a login shell on `donphan` after cloning the repository: ```bash # 1. Connect via the web portal → HPC Login → Interactive Apps > Shell (tmux) # Set cluster to "donphan (interactive/debug)" -# 2. Clone the project (if not done yet) +# 2. Clone the project into $VSC_HOME (if not done yet) +cd $VSC_HOME git clone cd 2026SEL3-project-BrittleStar @@ -24,29 +56,45 @@ cd 2026SEL3-project-BrittleStar bash scripts/hpc/install.sh ``` -> **Note:** virtual environments are cluster-specific. Re-run the script if you switch to a new cluster. +This uses the official [`vsc-venv`](https://docs.hpc.ugent.be/Linux/setting_up_python_virtual_environments/#vsc-venv-python-virtual-environment-wrapper-script) wrapper to: +- Redirect caches to `$VSC_SCRATCH` (to preserve your `$VSC_HOME` quota) +- Load the EasyBuild modules listed in `env/hpc/modules.txt` (JAX, Flax, WandB, …) +- Create a per-cluster virtual environment in `$VSC_DATA` +- Pip-install the remaining packages from `env/hpc/requirements.txt` +- Register a Jupyter kernel named `SEL3 ()` + +> **Note:** virtual environments are cluster-specific. Re-run the script when switching to a new cluster. + +--- ## Interactive Debugging on donphan -The `donphan` cluster provides quick access to a fractional A100 GPU and is ideal for verifying your environment end-to-end before submitting batch jobs. +The `donphan` cluster provides quick access and is ideal for verifying your environment before submitting batch jobs. ### Option A — Interactive shell session ```bash -# Swap to the debug cluster (from the login node) +# Swap to the debug cluster (from any login node) module swap cluster/donphan # Request an interactive job (1 node, 4 cores) qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00 -# Once inside the job, activate your environment -cd "$PBS_O_WORKDIR" # or cd to your project root if not set +# Once inside the job — redirect caches to scratch first +export XDG_CACHE_HOME="$VSC_SCRATCH/.cache" +export UV_CACHE_DIR="$VSC_SCRATCH/.cache/uv" +export MPLCONFIGDIR="$VSC_SCRATCH/.config/matplotlib" + +# Change to the project directory and activate environment +cd "$PBS_O_WORKDIR" module load vsc-venv source vsc-venv --activate --modules env/hpc/modules.txt -# Run the smoke test +# Set headless rendering backend export MUJOCO_GL=egl -python src/train.py --config configs/hpc_smoke_test.yaml + +# Run the smoke test +python src/train.py --config configs/hpc/smoke_test.yaml ``` ### Option B — JupyterLab session (HPC web portal) @@ -54,13 +102,13 @@ python src/train.py --config configs/hpc_smoke_test.yaml 1. In the web portal go to **Interactive Apps → JupyterLab RHEL9** 2. Set the following options: - | Option | Value | - |---------------------|---------------------------------| - | Cluster | `donphan (interactive/debug)` | - | Number of nodes | 1 | - | Number of cores | 4 | - | JupyterLab version | `4.2.5 GCCcore-13.3.0` | - | Custom code | *(leave blank — vsc-venv handles modules)* | + | Option | Value | + |---------------------|-------------------------------------------------| + | Cluster | `donphan (interactive/debug)` | + | Number of nodes | 1 | + | Number of cores | 4 | + | JupyterLab version | `4.2.5 GCCcore-13.3.0` | + | Custom code | *(leave blank — vsc-venv handles modules)* | 3. Click **Launch**, wait for the session to start, then **Connect**. 4. In JupyterLab, select the kernel **`SEL3 ()`**. @@ -74,6 +122,8 @@ python src/train.py --config configs/hpc_smoke_test.yaml > **Warning:** JAX can only be loaded by one kernel at a time. Shut down other kernels before switching notebooks. +--- + ## Submitting Batch Training Jobs ```bash @@ -85,6 +135,12 @@ module swap cluster/accelgor qsub scripts/hpc/train.pbs ``` +The job script automatically: +- Redirects all caches to `$VSC_SCRATCH` +- Writes run outputs to `$VSC_SCRATCH/runs/` during the run +- Copies final results to `$VSC_DATA/runs/` on completion +- Writes PBS stdout/stderr to `$VSC_DATA/runs//job.out` / `job.err` + Monitor your jobs: ```bash @@ -93,13 +149,11 @@ qstat -f # detailed info for a specific job qdel # cancel a job ``` -Logs are written to `runs/pbs_.out` and `runs/pbs_.err`. - --- ## Managing Dependencies -`env/hpc/requirements.txt` is the **source of truth** for pip packages on the HPC. It is automatically regenerated by CI whenever `pyproject.toml` changes. To regenerate locally: +`env/hpc/requirements.txt` is auto-generated by CI whenever `pyproject.toml` changes. To regenerate locally: ```bash uv run scripts/export_hpc_requirements.py