# HPC Guide (Ghent University Tier-2)

Full documentation: <https://docs.hpc.ugent.be/>

## Cluster Selection

Choose the appropriate cluster before submitting a job with `module swap cluster/<name>`. The default login cluster is **doduo**.

| Cluster   | Type                    | Use case                                      |
|-----------|-------------------------|-----------------------------------------------|
| `donphan` | Interactive / debug GPU | First-time setup, interactive debugging       |
| `doduo`   | CPU (default login)     | Rapid iteration, CPU-only smoke tests         |
| `joltik`  | GPU (A100 ¼-slice)      | Standard training runs                        |
| `accelgor`| GPU (A100 full)         | Large-scale / long experiments                |
| `litleo`  | GPU                     | Alternative GPU option                        |

> **Rule:** use at most **1 GPU per group at a time** on shared GPU clusters.  
> Check the current queue load at <https://shieldon.ugent.be:8083/pbsmon-web-users/>.

---

## Initial Environment Setup

Run **once** from a login node after cloning the repository:

```bash
# 1. Connect via the web portal → HPC Login → Interactive Apps > Shell (tmux)
#    Set cluster to "donphan (interactive/debug)"

# 2. Clone the project (if not done yet)
git clone <repo-url>
cd 2026SEL3-project-BrittleStar

# 3. Run the install script
bash scripts/hpc/install.sh
```

This uses the official [`vsc-venv`](https://docs.hpc.ugent.be/Linux/setting_up_python_virtual_environments/#vsc-venv-python-virtual-environment-wrapper-script) wrapper to:
- Load the EasyBuild modules listed in `env/hpc/modules.txt` (JAX, Flax, WandB, …)
- Create a per-cluster virtual environment in `$VSC_DATA`
- Pip-install the remaining packages from `env/hpc/requirements.txt`
- Register a Jupyter kernel named `sel3_<cluster>`

> **Note:** virtual environments are cluster-specific. Re-run the script if you switch to a new cluster.

---

## Interactive Debugging on donphan

The `donphan` cluster provides quick access to a fractional A100 GPU and is ideal for verifying your environment end-to-end before submitting batch jobs.

### Option A — Interactive shell session

```bash
# Swap to the debug cluster (from the login node)
module swap cluster/donphan

# Request an interactive job (1 node, 4 cores)
qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00

# Once inside the job, activate your environment
cd "$PBS_O_WORKDIR"   # or cd to your project root if not set
module load vsc-venv
source vsc-venv --activate --modules env/hpc/modules.txt

# Run the smoke test
export MUJOCO_GL=egl
python src/train.py --config configs/hpc_smoke_test.yaml
```

### Option B — JupyterLab session (HPC web portal)

1. In the web portal go to **Interactive Apps → JupyterLab RHEL9**
2. Set the following options:

   | Option              | Value                           |
   |---------------------|---------------------------------|
   | Cluster             | `donphan (interactive/debug)`   |
   | Number of nodes     | 1                               |
   | Number of cores     | 4                               |
   | JupyterLab version  | `4.2.5 GCCcore-13.3.0`          |
   | Custom code         | *(leave blank — vsc-venv handles modules)* |

3. Click **Launch**, wait for the session to start, then **Connect**.
4. In JupyterLab, select the kernel **`SEL3 (<cluster>)`**.
5. Verify GPU access:

   ```python
   import jax
   print(jax.default_backend())  # expected: 'gpu'
   print(jax.devices())          # expected: [CudaDevice(id=0)]
   ```

> **Warning:** JAX can only be loaded by one kernel at a time. Shut down other kernels before switching notebooks.

---

## Submitting Batch Training Jobs

```bash
# Default cluster (joltik — one A100 GPU slice)
qsub scripts/hpc/train.pbs

# Switch to a different GPU cluster first
module swap cluster/accelgor
qsub scripts/hpc/train.pbs
```

Monitor your jobs:

```bash
qstat          # list your jobs
qstat -f <id>  # detailed info for a specific job
qdel <id>      # cancel a job
```

Logs are written to `runs/pbs_<jobid>.out` and `runs/pbs_<jobid>.err`.

---

## Managing Dependencies

`env/hpc/requirements.txt` is the **source of truth** for pip packages on the HPC. It is automatically regenerated by CI whenever `pyproject.toml` changes. To regenerate locally:

```bash
uv run scripts/export_hpc_requirements.py
```

Do **not** edit `env/hpc/requirements.txt` by hand — edit `pyproject.toml` instead.