diff --git a/docs/HPC.md b/docs/HPC.md index a8ad9fc..f33d512 100644 --- a/docs/HPC.md +++ b/docs/HPC.md @@ -13,13 +13,37 @@ Run **once** after cloning the repository. This script handles all modules, mirr ```bash # Option A: Interactive (on a compute node) -module swap cluster/donphan # or joltik -qsub -I -l nodes=1:gpus=1 +module swap cluster/donphan # Debug cluster (CPU only) +# OR for GPU clusters: +# module swap cluster/joltik +# module swap cluster/accelgor +# module swap cluster/litleo + +qsub -I -l nodes=1:gpus=1 # Only for GPU clusters cd "${PBS_O_WORKDIR}" bash scripts/hpc/install.sh # Option B: Batch (Run in background) -qsub scripts/hpc/install.sh +# NOTE: GPU clusters (joltik/accelgor/litleo) require -l gpus=1 at runtime +qsub -l gpus=1 scripts/hpc/install.sh +``` + +## Production vs. Debug Clusters + +Our scripts are cluster-agnostic and do **not** have hardcoded GPU requirements. Instead, you must request GPUs at runtime using the `-l gpus=1` flag when submitting to a production GPU cluster. + +### Debugging (Donphan) +The `donphan` cluster does not support GPUs. Simply run the scripts without extra resource flags: +```bash +module swap cluster/donphan +qsub scripts/hpc/train.pbs +``` + +### Production (Joltik, Accelgor, Litleo) +These clusters provide GPU acceleration and **require** a GPU request at runtime: +```bash +module swap cluster/joltik # or accelgor/litleo +qsub -l gpus=1 scripts/hpc/train.pbs ``` ## Interactive Debugging diff --git a/scripts/hpc/install.sh b/scripts/hpc/install.sh index 2c4d12c..8a625a9 100644 --- a/scripts/hpc/install.sh +++ b/scripts/hpc/install.sh @@ -8,7 +8,7 @@ # qsub scripts/hpc/install.sh #PBS -N brittlestar-install -#PBS -l walltime=01:00:00 +#PBS -l walltime=00:15:00 set -euo pipefail diff --git a/scripts/hpc/train.pbs b/scripts/hpc/train.pbs index b4ae77c..3d53193 100644 --- a/scripts/hpc/train.pbs +++ b/scripts/hpc/train.pbs @@ -1,13 +1,11 @@ -#!/bin/bash -# scripts/hpc/train.pbs -# -# Usage (from project root): +# Production training (requires GPU at runtime): +# qsub -l gpus=1 scripts/hpc/train.pbs +# Debug/CPU training: # qsub scripts/hpc/train.pbs #PBS -N brittlestar-ppo -#PBS -l nodes=1:ppn=8:gpus=1 +#PBS -l nodes=1:ppn=8 #PBS -l walltime=24:00:00 -#PBS -l mem=32gb #PBS -o runs/brittlestar-ppo.o$PBS_JOBID #PBS -e runs/brittlestar-ppo.e$PBS_JOBID