docs(hpc): add docs/HPC.md and simplify README
- docs/HPC.md: cluster table, setup with vsc-venv, interactive debug (shell + JupyterLab on donphan), PBS batch submission, dependency mgmt - README.md: HPC section condensed to a single link to docs/HPC.md
This commit is contained in:
parent
9dce017525
commit
953e102483
2 changed files with 131 additions and 22 deletions
129
docs/HPC.md
Normal file
129
docs/HPC.md
Normal file
|
|
@ -0,0 +1,129 @@
|
|||
# HPC Guide (Ghent University Tier-2)
|
||||
|
||||
Full documentation: <https://docs.hpc.ugent.be/>
|
||||
|
||||
## Cluster Selection
|
||||
|
||||
Choose the appropriate cluster before submitting a job with `module swap cluster/<name>`. The default login cluster is **doduo**.
|
||||
|
||||
| Cluster | Type | Use case |
|
||||
|-----------|-------------------------|-----------------------------------------------|
|
||||
| `donphan` | Interactive / debug GPU | First-time setup, interactive debugging |
|
||||
| `doduo` | CPU (default login) | Rapid iteration, CPU-only smoke tests |
|
||||
| `joltik` | GPU (A100 ¼-slice) | Standard training runs |
|
||||
| `accelgor`| GPU (A100 full) | Large-scale / long experiments |
|
||||
| `litleo` | GPU | Alternative GPU option |
|
||||
|
||||
> **Rule:** use at most **1 GPU per group at a time** on shared GPU clusters.
|
||||
> Check the current queue load at <https://shieldon.ugent.be:8083/pbsmon-web-users/>.
|
||||
|
||||
---
|
||||
|
||||
## Initial Environment Setup
|
||||
|
||||
Run **once** from a login node after cloning the repository:
|
||||
|
||||
```bash
|
||||
# 1. Connect via the web portal → HPC Login → Interactive Apps > Shell (tmux)
|
||||
# Set cluster to "donphan (interactive/debug)"
|
||||
|
||||
# 2. Clone the project (if not done yet)
|
||||
git clone <repo-url>
|
||||
cd 2026SEL3-project-BrittleStar
|
||||
|
||||
# 3. Run the install script
|
||||
bash scripts/hpc/install.sh
|
||||
```
|
||||
|
||||
This uses the official [`vsc-venv`](https://docs.hpc.ugent.be/Linux/setting_up_python_virtual_environments/#vsc-venv-python-virtual-environment-wrapper-script) wrapper to:
|
||||
- Load the EasyBuild modules listed in `env/hpc/modules.txt` (JAX, Flax, WandB, …)
|
||||
- Create a per-cluster virtual environment in `$VSC_DATA`
|
||||
- Pip-install the remaining packages from `env/hpc/requirements.txt`
|
||||
- Register a Jupyter kernel named `sel3_<cluster>`
|
||||
|
||||
> **Note:** virtual environments are cluster-specific. Re-run the script if you switch to a new cluster.
|
||||
|
||||
---
|
||||
|
||||
## Interactive Debugging on donphan
|
||||
|
||||
The `donphan` cluster provides quick access to a fractional A100 GPU and is ideal for verifying your environment end-to-end before submitting batch jobs.
|
||||
|
||||
### Option A — Interactive shell session
|
||||
|
||||
```bash
|
||||
# Swap to the debug cluster (from the login node)
|
||||
module swap cluster/donphan
|
||||
|
||||
# Request an interactive job (1 node, 4 cores)
|
||||
qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00
|
||||
|
||||
# Once inside the job, activate your environment
|
||||
cd "$PBS_O_WORKDIR" # or cd to your project root if not set
|
||||
module load vsc-venv
|
||||
source vsc-venv --activate --modules env/hpc/modules.txt
|
||||
|
||||
# Run the smoke test
|
||||
export MUJOCO_GL=egl
|
||||
python src/train.py --config configs/hpc_smoke_test.yaml
|
||||
```
|
||||
|
||||
### Option B — JupyterLab session (HPC web portal)
|
||||
|
||||
1. In the web portal go to **Interactive Apps → JupyterLab RHEL9**
|
||||
2. Set the following options:
|
||||
|
||||
| Option | Value |
|
||||
|---------------------|---------------------------------|
|
||||
| Cluster | `donphan (interactive/debug)` |
|
||||
| Number of nodes | 1 |
|
||||
| Number of cores | 4 |
|
||||
| JupyterLab version | `4.2.5 GCCcore-13.3.0` |
|
||||
| Custom code | *(leave blank — vsc-venv handles modules)* |
|
||||
|
||||
3. Click **Launch**, wait for the session to start, then **Connect**.
|
||||
4. In JupyterLab, select the kernel **`SEL3 (<cluster>)`**.
|
||||
5. Verify GPU access:
|
||||
|
||||
```python
|
||||
import jax
|
||||
print(jax.default_backend()) # expected: 'gpu'
|
||||
print(jax.devices()) # expected: [CudaDevice(id=0)]
|
||||
```
|
||||
|
||||
> **Warning:** JAX can only be loaded by one kernel at a time. Shut down other kernels before switching notebooks.
|
||||
|
||||
---
|
||||
|
||||
## Submitting Batch Training Jobs
|
||||
|
||||
```bash
|
||||
# Default cluster (joltik — one A100 GPU slice)
|
||||
qsub scripts/hpc/train.pbs
|
||||
|
||||
# Switch to a different GPU cluster first
|
||||
module swap cluster/accelgor
|
||||
qsub scripts/hpc/train.pbs
|
||||
```
|
||||
|
||||
Monitor your jobs:
|
||||
|
||||
```bash
|
||||
qstat # list your jobs
|
||||
qstat -f <id> # detailed info for a specific job
|
||||
qdel <id> # cancel a job
|
||||
```
|
||||
|
||||
Logs are written to `runs/pbs_<jobid>.out` and `runs/pbs_<jobid>.err`.
|
||||
|
||||
---
|
||||
|
||||
## Managing Dependencies
|
||||
|
||||
`env/hpc/requirements.txt` is the **source of truth** for pip packages on the HPC. It is automatically regenerated by CI whenever `pyproject.toml` changes. To regenerate locally:
|
||||
|
||||
```bash
|
||||
uv run scripts/export_hpc_requirements.py
|
||||
```
|
||||
|
||||
Do **not** edit `env/hpc/requirements.txt` by hand — edit `pyproject.toml` instead.
|
||||
Reference in a new issue