refactor: comment cleanups
This commit is contained in:
parent
953e102483
commit
08d54b73b4
3 changed files with 1 additions and 37 deletions
23
docs/HPC.md
23
docs/HPC.md
|
|
@ -1,4 +1,4 @@
|
||||||
# HPC Guide (Ghent University Tier-2)
|
# HPC Guide
|
||||||
|
|
||||||
Full documentation: <https://docs.hpc.ugent.be/>
|
Full documentation: <https://docs.hpc.ugent.be/>
|
||||||
|
|
||||||
|
|
@ -6,19 +6,8 @@ Full documentation: <https://docs.hpc.ugent.be/>
|
||||||
|
|
||||||
Choose the appropriate cluster before submitting a job with `module swap cluster/<name>`. The default login cluster is **doduo**.
|
Choose the appropriate cluster before submitting a job with `module swap cluster/<name>`. The default login cluster is **doduo**.
|
||||||
|
|
||||||
| Cluster | Type | Use case |
|
|
||||||
|-----------|-------------------------|-----------------------------------------------|
|
|
||||||
| `donphan` | Interactive / debug GPU | First-time setup, interactive debugging |
|
|
||||||
| `doduo` | CPU (default login) | Rapid iteration, CPU-only smoke tests |
|
|
||||||
| `joltik` | GPU (A100 ¼-slice) | Standard training runs |
|
|
||||||
| `accelgor`| GPU (A100 full) | Large-scale / long experiments |
|
|
||||||
| `litleo` | GPU | Alternative GPU option |
|
|
||||||
|
|
||||||
> **Rule:** use at most **1 GPU per group at a time** on shared GPU clusters.
|
|
||||||
> Check the current queue load at <https://shieldon.ugent.be:8083/pbsmon-web-users/>.
|
> Check the current queue load at <https://shieldon.ugent.be:8083/pbsmon-web-users/>.
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Initial Environment Setup
|
## Initial Environment Setup
|
||||||
|
|
||||||
Run **once** from a login node after cloning the repository:
|
Run **once** from a login node after cloning the repository:
|
||||||
|
|
@ -35,16 +24,8 @@ cd 2026SEL3-project-BrittleStar
|
||||||
bash scripts/hpc/install.sh
|
bash scripts/hpc/install.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
This uses the official [`vsc-venv`](https://docs.hpc.ugent.be/Linux/setting_up_python_virtual_environments/#vsc-venv-python-virtual-environment-wrapper-script) wrapper to:
|
|
||||||
- Load the EasyBuild modules listed in `env/hpc/modules.txt` (JAX, Flax, WandB, …)
|
|
||||||
- Create a per-cluster virtual environment in `$VSC_DATA`
|
|
||||||
- Pip-install the remaining packages from `env/hpc/requirements.txt`
|
|
||||||
- Register a Jupyter kernel named `sel3_<cluster>`
|
|
||||||
|
|
||||||
> **Note:** virtual environments are cluster-specific. Re-run the script if you switch to a new cluster.
|
> **Note:** virtual environments are cluster-specific. Re-run the script if you switch to a new cluster.
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Interactive Debugging on donphan
|
## Interactive Debugging on donphan
|
||||||
|
|
||||||
The `donphan` cluster provides quick access to a fractional A100 GPU and is ideal for verifying your environment end-to-end before submitting batch jobs.
|
The `donphan` cluster provides quick access to a fractional A100 GPU and is ideal for verifying your environment end-to-end before submitting batch jobs.
|
||||||
|
|
@ -93,8 +74,6 @@ python src/train.py --config configs/hpc_smoke_test.yaml
|
||||||
|
|
||||||
> **Warning:** JAX can only be loaded by one kernel at a time. Shut down other kernels before switching notebooks.
|
> **Warning:** JAX can only be loaded by one kernel at a time. Shut down other kernels before switching notebooks.
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Submitting Batch Training Jobs
|
## Submitting Batch Training Jobs
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
|
||||||
|
|
@ -12,18 +12,12 @@
|
||||||
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
echo "==> Loading vsc-venv module..."
|
|
||||||
module load vsc-venv
|
module load vsc-venv
|
||||||
|
|
||||||
echo "==> Activating virtual environment (created per-cluster in \$VSC_DATA)..."
|
|
||||||
# vsc-venv transparently creates and manages a per-cluster venv in $VSC_DATA,
|
|
||||||
# loading the modules listed in env/hpc/modules.txt and pip-installing anything
|
|
||||||
# in env/hpc/requirements.txt that is not already satisfied.
|
|
||||||
source vsc-venv --activate \
|
source vsc-venv --activate \
|
||||||
--modules env/hpc/modules.txt \
|
--modules env/hpc/modules.txt \
|
||||||
--requirements env/hpc/requirements.txt
|
--requirements env/hpc/requirements.txt
|
||||||
|
|
||||||
echo "==> Registering Jupyter kernel..."
|
|
||||||
python -m ipykernel install --user --name="sel3_${VSC_INSTITUTE_CLUSTER}" \
|
python -m ipykernel install --user --name="sel3_${VSC_INSTITUTE_CLUSTER}" \
|
||||||
--display-name "SEL3 (${VSC_INSTITUTE_CLUSTER})"
|
--display-name "SEL3 (${VSC_INSTITUTE_CLUSTER})"
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -6,27 +6,18 @@
|
||||||
#
|
#
|
||||||
# To target a specific GPU cluster (default: joltik):
|
# To target a specific GPU cluster (default: joltik):
|
||||||
# module swap cluster/accelgor && qsub scripts/hpc/train.pbs
|
# module swap cluster/accelgor && qsub scripts/hpc/train.pbs
|
||||||
#
|
|
||||||
# Available GPU clusters: joltik, accelgor, litleo
|
|
||||||
# Debug / interactive: doduo (CPU) or donphan (interactive GPU)
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# PBS job directives
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
#PBS -N brittlestar-ppo
|
#PBS -N brittlestar-ppo
|
||||||
#PBS -l nodes=1:ppn=8:gpus=1
|
#PBS -l nodes=1:ppn=8:gpus=1
|
||||||
#PBS -l walltime=24:00:00
|
#PBS -l walltime=24:00:00
|
||||||
#PBS -l mem=32gb
|
#PBS -l mem=32gb
|
||||||
#PBS -o runs/pbs_${PBS_JOBID}.out
|
#PBS -o runs/pbs_${PBS_JOBID}.out
|
||||||
#PBS -e runs/pbs_${PBS_JOBID}.err
|
#PBS -e runs/pbs_${PBS_JOBID}.err
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
cd "$PBS_O_WORKDIR"
|
cd "$PBS_O_WORKDIR"
|
||||||
|
|
||||||
# Activate the project virtual environment via the official vsc-venv wrapper.
|
|
||||||
# This loads the modules listed in env/hpc/modules.txt and the per-cluster venv.
|
|
||||||
module load vsc-venv
|
module load vsc-venv
|
||||||
source vsc-venv --activate --modules env/hpc/modules.txt
|
source vsc-venv --activate --modules env/hpc/modules.txt
|
||||||
|
|
||||||
|
|
|
||||||
Reference in a new issue