108 lines
3.2 KiB
Markdown
108 lines
3.2 KiB
Markdown
# HPC Guide
|
|
|
|
Full documentation: <https://docs.hpc.ugent.be/>
|
|
|
|
## Cluster Selection
|
|
|
|
Choose the appropriate cluster before submitting a job with `module swap cluster/<name>`. The default login cluster is **doduo**.
|
|
|
|
> Check the current queue load at <https://shieldon.ugent.be:8083/pbsmon-web-users/>.
|
|
|
|
## Initial Environment Setup
|
|
|
|
Run **once** from a login node after cloning the repository:
|
|
|
|
```bash
|
|
# 1. Connect via the web portal → HPC Login → Interactive Apps > Shell (tmux)
|
|
# Set cluster to "donphan (interactive/debug)"
|
|
|
|
# 2. Clone the project (if not done yet)
|
|
git clone <repo-url>
|
|
cd 2026SEL3-project-BrittleStar
|
|
|
|
# 3. Run the install script
|
|
bash scripts/hpc/install.sh
|
|
```
|
|
|
|
> **Note:** virtual environments are cluster-specific. Re-run the script if you switch to a new cluster.
|
|
|
|
## Interactive Debugging on donphan
|
|
|
|
The `donphan` cluster provides quick access to a fractional A100 GPU and is ideal for verifying your environment end-to-end before submitting batch jobs.
|
|
|
|
### Option A — Interactive shell session
|
|
|
|
```bash
|
|
# Swap to the debug cluster (from the login node)
|
|
module swap cluster/donphan
|
|
|
|
# Request an interactive job (1 node, 4 cores)
|
|
qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00
|
|
|
|
# Once inside the job, activate your environment
|
|
cd "$PBS_O_WORKDIR" # or cd to your project root if not set
|
|
module load vsc-venv
|
|
source vsc-venv --activate --modules env/hpc/modules.txt
|
|
|
|
# Run the smoke test
|
|
export MUJOCO_GL=egl
|
|
python src/train.py --config configs/hpc_smoke_test.yaml
|
|
```
|
|
|
|
### Option B — JupyterLab session (HPC web portal)
|
|
|
|
1. In the web portal go to **Interactive Apps → JupyterLab RHEL9**
|
|
2. Set the following options:
|
|
|
|
| Option | Value |
|
|
|---------------------|---------------------------------|
|
|
| Cluster | `donphan (interactive/debug)` |
|
|
| Number of nodes | 1 |
|
|
| Number of cores | 4 |
|
|
| JupyterLab version | `4.2.5 GCCcore-13.3.0` |
|
|
| Custom code | *(leave blank — vsc-venv handles modules)* |
|
|
|
|
3. Click **Launch**, wait for the session to start, then **Connect**.
|
|
4. In JupyterLab, select the kernel **`SEL3 (<cluster>)`**.
|
|
5. Verify GPU access:
|
|
|
|
```python
|
|
import jax
|
|
print(jax.default_backend()) # expected: 'gpu'
|
|
print(jax.devices()) # expected: [CudaDevice(id=0)]
|
|
```
|
|
|
|
> **Warning:** JAX can only be loaded by one kernel at a time. Shut down other kernels before switching notebooks.
|
|
|
|
## Submitting Batch Training Jobs
|
|
|
|
```bash
|
|
# Default cluster (joltik — one A100 GPU slice)
|
|
qsub scripts/hpc/train.pbs
|
|
|
|
# Switch to a different GPU cluster first
|
|
module swap cluster/accelgor
|
|
qsub scripts/hpc/train.pbs
|
|
```
|
|
|
|
Monitor your jobs:
|
|
|
|
```bash
|
|
qstat # list your jobs
|
|
qstat -f <id> # detailed info for a specific job
|
|
qdel <id> # cancel a job
|
|
```
|
|
|
|
Logs are written to `runs/pbs_<jobid>.out` and `runs/pbs_<jobid>.err`.
|
|
|
|
---
|
|
|
|
## Managing Dependencies
|
|
|
|
`env/hpc/requirements.txt` is the **source of truth** for pip packages on the HPC. It is automatically regenerated by CI whenever `pyproject.toml` changes. To regenerate locally:
|
|
|
|
```bash
|
|
uv run scripts/export_hpc_requirements.py
|
|
```
|
|
|
|
Do **not** edit `env/hpc/requirements.txt` by hand — edit `pyproject.toml` instead.
|