diff --git a/docs/HPC.md b/docs/HPC.md index 81a7a82..a8ad9fc 100644 --- a/docs/HPC.md +++ b/docs/HPC.md @@ -9,29 +9,25 @@ Full documentation: ## Initial Environment Setup -Run **once** after cloning the repository. Ensure you are logged into a **compute node** on `donphan` or `joltik`. - -> [!IMPORTANT] -> To avoid the **3GB home directory quota limit**, the installation script mirrors your configuration files to **`$VSC_DATA`** (25GB+ quota). The `vsc-venv` tool then automatically creates and manages the environment on the larger partition. +Run **once** after cloning the repository. This script handles all modules, mirroring, and environment synchronization. ```bash -# 1. Start an interactive session (donphan for debug, joltik for training) -qsub -I -l nodes=1:ppn=8:gpus=1 - -# 2. Run the streamlined install script +# Option A: Interactive (on a compute node) +module swap cluster/donphan # or joltik +qsub -I -l nodes=1:gpus=1 cd "${PBS_O_WORKDIR}" bash scripts/hpc/install.sh + +# Option B: Batch (Run in background) +qsub scripts/hpc/install.sh ``` ## Interactive Debugging -You can use the same `install.sh` script to quickly activate your environment for interactive work. +To activate your environment for interactive work, simply run the same `install.sh` script. ```bash -# Request an interactive job qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00 - -# Change to project directory and run install.sh to sync and activate cd "$PBS_O_WORKDIR" bash scripts/hpc/install.sh ``` @@ -40,39 +36,39 @@ bash scripts/hpc/install.sh After installation, run these commands to ensure your environment is set up correctly: -1. **Verify Location**: +1. **Verify Quota Safety**: ```bash - # Confirm that NO 'venvs' folder appeared in your project root - ls -d venvs 2>/dev/null # Should return 'not found' - - # Confirm the environment is on the data partition - python -c "import torch; print(torch.__file__)" - # Expected: /kyukon/data/gent/vsc... or similar + ls -d venvs 2>/dev/null && echo "FAIL" || echo ">>> PASS: Project root is clean." + ``` +2. **Verify Library Versions (NumPy Fix)**: + ```bash + python -c "import numpy; print(f'NumPy: {numpy.__version__}')" + # Expected: 2.x.x (Venv version), not 1.2x (System version) + ``` +3. **Verify GPU Access**: + ```bash + python -c "import torch, jax; print(f'GPU: {torch.cuda.is_available()}'); print(f'JAX: {jax.devices()}')" ``` -2. **Verify GPU Access**: - ```bash - python -c "import torch; import jax; print(f'Torch CUDA: {torch.cuda.is_available()}'); print(f'JAX Devices: {jax.devices()}')" - ``` - *Expected output: `Torch CUDA: True` and `JAX Devices: [CudaDevice(id=0)]`.* +## PR Verification (Quick Start) -3. **Verify Home Quota**: - ```bash - df -h ~ # Should show low usage (< 1GB typically) - ``` - -## Submitting Batch Training Jobs +If you are a reviewer verifying a PR, run this single block: ```bash -# Submit to the default cluster (joltik) +git checkout +module swap cluster/donphan +qsub -I -l nodes=1:gpus=1 + +# Inside the interactive session: +cd "$PBS_O_WORKDIR" +bash scripts/hpc/install.sh +python -c "import torch, jax; print(torch.cuda.is_available()); print(jax.devices())" +exit + +# Verify batch submission qsub scripts/hpc/train.pbs - -# To choose a different cluster (e.g. donphan debug) without touching code -module swap cluster/donphan && qsub scripts/hpc/train.pbs ``` -The `train.pbs` script automatically handles its own activation using the mirrored configurations on `$VSC_DATA`. - ## Managing Dependencies `env/hpc/requirements.txt` is auto-generated from `pyproject.toml`. To regenerate: diff --git a/scripts/hpc/install.sh b/scripts/hpc/install.sh index a37e94a..328252c 100644 --- a/scripts/hpc/install.sh +++ b/scripts/hpc/install.sh @@ -1,11 +1,29 @@ #!/bin/bash -l # scripts/hpc/install.sh # -set -eo pipefail +# Usage (on any compute node): +# bash scripts/hpc/install.sh +# +# Batch usage: +# qsub scripts/hpc/install.sh -echo ">>> Starting HPC Installation in $(hostname)..." +#PBS -N brittlestar-install +#PBS -l nodes=1:ppn=1:gpus=1 +#PBS -l walltime=01:00:00 +#PBS -o scripts/hpc/install.o$PBS_JOBID +#PBS -e scripts/hpc/install.e$PBS_JOBID + +set -euo pipefail + +# Preliminary status echo +echo ">>> Starting installation job $PBS_JOBID on $(hostname)..." + +if [ -n "$PBS_O_WORKDIR" ]; then + cd "$PBS_O_WORKDIR" +fi # Mirror configs to $VSC_DATA to avoid home quota limits (3GB) +# vsc-venv manages environments relative to the requirements file PROJ_NAME=$(basename "$PWD") HPC_CONFIG_DIR="$VSC_DATA/$PROJ_NAME/env/hpc" mkdir -p "$HPC_CONFIG_DIR" @@ -14,13 +32,17 @@ cp env/hpc/*.txt "$HPC_CONFIG_DIR/" module load vsc-venv echo ">>> Synchronizing and activating environment (vsc-venv)..." -set +eo pipefail +set +euo pipefail source vsc-venv --activate \ --modules "$HPC_CONFIG_DIR/modules.txt" \ --requirements "$HPC_CONFIG_DIR/requirements.txt" -set -eo pipefail +set -euo pipefail -# Overlay specific NumPy/Protobuf versions to ensure venv precedence +# Force the venv path to the front of PYTHONPATH to override system modules (e.g. NumPy 1.2x) +VENV_LIB_DIR="$VIRTUAL_ENV/lib/python$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')/site-packages" +export PYTHONPATH="$VENV_LIB_DIR:$PYTHONPATH" + +# Overlay modern NumPy/Protobuf versions echo ">>> Applying library overlays (NumPy, Protobuf)..." pip install --upgrade --no-deps numpy protobuf diff --git a/scripts/hpc/train.pbs b/scripts/hpc/train.pbs index 83d6284..53d37c3 100644 --- a/scripts/hpc/train.pbs +++ b/scripts/hpc/train.pbs @@ -20,7 +20,7 @@ if [ -n "$PBS_O_WORKDIR" ]; then cd "$PBS_O_WORKDIR" fi -# Set up storage paths immediately +# Set up storage paths dynamically PROJ_NAME=$(basename "$PWD") RUN_ID="brittlestar_${PBS_JOBID}" SCRATCH_RUNDIR="$VSC_SCRATCH/runs/$RUN_ID" @@ -41,12 +41,17 @@ if [ ! -d "$HPC_CONFIG_DIR" ]; then exit 1 fi -set +eo pipefail +set +euo pipefail source vsc-venv --activate \ --modules "$HPC_CONFIG_DIR/modules.txt" \ --requirements "$HPC_CONFIG_DIR/requirements.txt" set -euo pipefail +# Force the venv path to the front of PYTHONPATH to override system modules (e.g. NumPy 1.2x) +# This is required because VSC system modules are appended to PYTHONPATH and would otherwise shadow your venv +VENV_LIB_DIR="$VIRTUAL_ENV/lib/python$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')/site-packages" +export PYTHONPATH="$VENV_LIB_DIR:$PYTHONPATH" + echo ">>> Starting BrittleStar training..." export MUJOCO_GL=egl export WANDB_DIR="$SCRATCH_RUNDIR"