2026SEL3-project-Brittle_Star_Controller_Modularity/docs/HPC.md

HPC Guide

Full documentation: https://docs.hpc.ugent.be/

Cluster Selection

Choose the appropriate cluster before submitting a job with module swap cluster/<name>. The default login cluster is doduo.

Cluster	Type	Use case
donphan	Interactive / debug GPU	First-time setup, interactive debugging
doduo	CPU (default login)	Rapid iteration, CPU-only smoke tests
joltik	GPU (A100 ¼-slice)	Standard training runs
accelgor	GPU (A100 full)	Large-scale / long experiments
litleo	GPU	Alternative GPU option

Rule: use at most 1 GPU per group at a time on shared GPU clusters.
Check the current queue load at https://shieldon.ugent.be:8083/pbsmon-web-users/.

---

Storage Overview

The HPC provides three filesystems for different purposes. Understanding this is critical to avoid filling up your home directory.

Variable	Typical size	Purpose
$VSC_HOME	~3 GB	Config files, SSH keys, project source code only
$VSC_DATA	~25 GB	Persistent outputs: trained models, final logs
$VSC_SCRATCH	Large	Fast I/O during jobs: caches, intermediate files

Important:

• Clone the repository into $VSC_HOME — it is small in size and accessible from all clusters.
• All caches (pip, uv, matplotlib) must be redirected to $VSC_SCRATCH to avoid filling $VSC_HOME.
• Run outputs are written to $VSC_SCRATCH during the job (fast I/O) and copied to $VSC_DATA at the end for persistence.
• $VSC_SCRATCH may be purged periodically — do not use it as long-term storage.

Check your quota: https://account.vscentrum.be (Usage section).

---

Initial Environment Setup

Run once from a login shell on donphan after cloning the repository:

# 1. Connect via the web portal → HPC Login → Interactive Apps > Shell (tmux)
#    Set cluster to "donphan (interactive/debug)"

# 2. Clone the project into $VSC_HOME (if not done yet)
cd $VSC_HOME
git clone <repo-url>
cd 2026SEL3-project-BrittleStar

# 3. Run the install script
bash scripts/hpc/install.sh

This uses the official vsc-venv wrapper to:

• Redirect caches to $VSC_SCRATCH (to preserve your $VSC_HOME quota)
• Load the EasyBuild modules listed in env/hpc/modules.txt (JAX, Flax, WandB, …)
• Create a per-cluster virtual environment in $VSC_DATA
• Pip-install the remaining packages from env/hpc/requirements.txt
• Register a Jupyter kernel named SEL3 (<cluster>)

Note: virtual environments are cluster-specific. Re-run the script when switching to a new cluster.

---

Interactive Debugging on donphan

The donphan cluster provides quick access and is ideal for verifying your environment before submitting batch jobs.

Option A — Interactive shell session

# Swap to the debug cluster (from any login node)
module swap cluster/donphan

# Request an interactive job (1 node, 4 cores)
qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00

# Once inside the job — redirect caches to scratch first
export PIP_CACHE_DIR="$VSC_SCRATCH/.cache/pip"
export UV_CACHE_DIR="$VSC_SCRATCH/.cache/uv"

# Change to the project directory and activate environment
cd "$PBS_O_WORKDIR"
module load vsc-venv
source vsc-venv --activate --modules env/hpc/modules.txt

# Set headless rendering backend
export MUJOCO_GL=egl

# Run the smoke test
python src/train.py --config configs/hpc/smoke_test.yaml

Option B — JupyterLab session (HPC web portal)

1. In the web portal go to Interactive Apps → JupyterLab RHEL9

2. Set the following options:

   Option	Value
   Cluster	donphan (interactive/debug)
   Number of nodes	1
   Number of cores	4
   JupyterLab version	4.2.5 GCCcore-13.3.0
   Custom code	(leave blank — vsc-venv handles modules)

3. Click Launch, wait for the session to start, then Connect.

4. In JupyterLab, select the kernel SEL3 (<cluster>).

5. Verify GPU access:

   import jax
   print(jax.default_backend())  # expected: 'gpu'
   print(jax.devices())          # expected: [CudaDevice(id=0)]
   

Warning: JAX can only be loaded by one kernel at a time. Shut down other kernels before switching notebooks.

---

Submitting Batch Training Jobs

# Default cluster (joltik — one A100 GPU slice)
qsub scripts/hpc/train.pbs

# Switch to a different GPU cluster first
module swap cluster/accelgor
qsub scripts/hpc/train.pbs

The job script automatically:

• Redirects all caches to $VSC_SCRATCH
• Writes run outputs to $VSC_SCRATCH/runs/<job_id> during the run
• Copies final results to $VSC_DATA/runs/<job_id> on completion
• Writes PBS stdout/stderr to $VSC_DATA/runs/<job_id>/job.out / job.err

Monitor your jobs:

qstat          # list your jobs
qstat -f <id>  # detailed info for a specific job
qdel <id>      # cancel a job

---

Managing Dependencies

env/hpc/requirements.txt is auto-generated by CI whenever pyproject.toml changes. To regenerate locally:

uv run scripts/export_hpc_requirements.py

Do not edit env/hpc/requirements.txt by hand — edit pyproject.toml instead.