diff --git a/docs/HPC.md b/docs/HPC.md
index 81a7a82..a8ad9fc 100644
--- a/docs/HPC.md
+++ b/docs/HPC.md
@@ -9,29 +9,25 @@ Full documentation:
## Initial Environment Setup
-Run **once** after cloning the repository. Ensure you are logged into a **compute node** on `donphan` or `joltik`.
-
-> [!IMPORTANT]
-> To avoid the **3GB home directory quota limit**, the installation script mirrors your configuration files to **`$VSC_DATA`** (25GB+ quota). The `vsc-venv` tool then automatically creates and manages the environment on the larger partition.
+Run **once** after cloning the repository. This script handles all modules, mirroring, and environment synchronization.
```bash
-# 1. Start an interactive session (donphan for debug, joltik for training)
-qsub -I -l nodes=1:ppn=8:gpus=1
-
-# 2. Run the streamlined install script
+# Option A: Interactive (on a compute node)
+module swap cluster/donphan # or joltik
+qsub -I -l nodes=1:gpus=1
cd "${PBS_O_WORKDIR}"
bash scripts/hpc/install.sh
+
+# Option B: Batch (Run in background)
+qsub scripts/hpc/install.sh
```
## Interactive Debugging
-You can use the same `install.sh` script to quickly activate your environment for interactive work.
+To activate your environment for interactive work, simply run the same `install.sh` script.
```bash
-# Request an interactive job
qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00
-
-# Change to project directory and run install.sh to sync and activate
cd "$PBS_O_WORKDIR"
bash scripts/hpc/install.sh
```
@@ -40,39 +36,39 @@ bash scripts/hpc/install.sh
After installation, run these commands to ensure your environment is set up correctly:
-1. **Verify Location**:
+1. **Verify Quota Safety**:
```bash
- # Confirm that NO 'venvs' folder appeared in your project root
- ls -d venvs 2>/dev/null # Should return 'not found'
-
- # Confirm the environment is on the data partition
- python -c "import torch; print(torch.__file__)"
- # Expected: /kyukon/data/gent/vsc... or similar
+ ls -d venvs 2>/dev/null && echo "FAIL" || echo ">>> PASS: Project root is clean."
+ ```
+2. **Verify Library Versions (NumPy Fix)**:
+ ```bash
+ python -c "import numpy; print(f'NumPy: {numpy.__version__}')"
+ # Expected: 2.x.x (Venv version), not 1.2x (System version)
+ ```
+3. **Verify GPU Access**:
+ ```bash
+ python -c "import torch, jax; print(f'GPU: {torch.cuda.is_available()}'); print(f'JAX: {jax.devices()}')"
```
-2. **Verify GPU Access**:
- ```bash
- python -c "import torch; import jax; print(f'Torch CUDA: {torch.cuda.is_available()}'); print(f'JAX Devices: {jax.devices()}')"
- ```
- *Expected output: `Torch CUDA: True` and `JAX Devices: [CudaDevice(id=0)]`.*
+## PR Verification (Quick Start)
-3. **Verify Home Quota**:
- ```bash
- df -h ~ # Should show low usage (< 1GB typically)
- ```
-
-## Submitting Batch Training Jobs
+If you are a reviewer verifying a PR, run this single block:
```bash
-# Submit to the default cluster (joltik)
+git checkout
+module swap cluster/donphan
+qsub -I -l nodes=1:gpus=1
+
+# Inside the interactive session:
+cd "$PBS_O_WORKDIR"
+bash scripts/hpc/install.sh
+python -c "import torch, jax; print(torch.cuda.is_available()); print(jax.devices())"
+exit
+
+# Verify batch submission
qsub scripts/hpc/train.pbs
-
-# To choose a different cluster (e.g. donphan debug) without touching code
-module swap cluster/donphan && qsub scripts/hpc/train.pbs
```
-The `train.pbs` script automatically handles its own activation using the mirrored configurations on `$VSC_DATA`.
-
## Managing Dependencies
`env/hpc/requirements.txt` is auto-generated from `pyproject.toml`. To regenerate:
diff --git a/scripts/hpc/install.sh b/scripts/hpc/install.sh
index a37e94a..328252c 100644
--- a/scripts/hpc/install.sh
+++ b/scripts/hpc/install.sh
@@ -1,11 +1,29 @@
#!/bin/bash -l
# scripts/hpc/install.sh
#
-set -eo pipefail
+# Usage (on any compute node):
+# bash scripts/hpc/install.sh
+#
+# Batch usage:
+# qsub scripts/hpc/install.sh
-echo ">>> Starting HPC Installation in $(hostname)..."
+#PBS -N brittlestar-install
+#PBS -l nodes=1:ppn=1:gpus=1
+#PBS -l walltime=01:00:00
+#PBS -o scripts/hpc/install.o$PBS_JOBID
+#PBS -e scripts/hpc/install.e$PBS_JOBID
+
+set -euo pipefail
+
+# Preliminary status echo
+echo ">>> Starting installation job $PBS_JOBID on $(hostname)..."
+
+if [ -n "$PBS_O_WORKDIR" ]; then
+ cd "$PBS_O_WORKDIR"
+fi
# Mirror configs to $VSC_DATA to avoid home quota limits (3GB)
+# vsc-venv manages environments relative to the requirements file
PROJ_NAME=$(basename "$PWD")
HPC_CONFIG_DIR="$VSC_DATA/$PROJ_NAME/env/hpc"
mkdir -p "$HPC_CONFIG_DIR"
@@ -14,13 +32,17 @@ cp env/hpc/*.txt "$HPC_CONFIG_DIR/"
module load vsc-venv
echo ">>> Synchronizing and activating environment (vsc-venv)..."
-set +eo pipefail
+set +euo pipefail
source vsc-venv --activate \
--modules "$HPC_CONFIG_DIR/modules.txt" \
--requirements "$HPC_CONFIG_DIR/requirements.txt"
-set -eo pipefail
+set -euo pipefail
-# Overlay specific NumPy/Protobuf versions to ensure venv precedence
+# Force the venv path to the front of PYTHONPATH to override system modules (e.g. NumPy 1.2x)
+VENV_LIB_DIR="$VIRTUAL_ENV/lib/python$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')/site-packages"
+export PYTHONPATH="$VENV_LIB_DIR:$PYTHONPATH"
+
+# Overlay modern NumPy/Protobuf versions
echo ">>> Applying library overlays (NumPy, Protobuf)..."
pip install --upgrade --no-deps numpy protobuf
diff --git a/scripts/hpc/train.pbs b/scripts/hpc/train.pbs
index 83d6284..53d37c3 100644
--- a/scripts/hpc/train.pbs
+++ b/scripts/hpc/train.pbs
@@ -20,7 +20,7 @@ if [ -n "$PBS_O_WORKDIR" ]; then
cd "$PBS_O_WORKDIR"
fi
-# Set up storage paths immediately
+# Set up storage paths dynamically
PROJ_NAME=$(basename "$PWD")
RUN_ID="brittlestar_${PBS_JOBID}"
SCRATCH_RUNDIR="$VSC_SCRATCH/runs/$RUN_ID"
@@ -41,12 +41,17 @@ if [ ! -d "$HPC_CONFIG_DIR" ]; then
exit 1
fi
-set +eo pipefail
+set +euo pipefail
source vsc-venv --activate \
--modules "$HPC_CONFIG_DIR/modules.txt" \
--requirements "$HPC_CONFIG_DIR/requirements.txt"
set -euo pipefail
+# Force the venv path to the front of PYTHONPATH to override system modules (e.g. NumPy 1.2x)
+# This is required because VSC system modules are appended to PYTHONPATH and would otherwise shadow your venv
+VENV_LIB_DIR="$VIRTUAL_ENV/lib/python$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')/site-packages"
+export PYTHONPATH="$VENV_LIB_DIR:$PYTHONPATH"
+
echo ">>> Starting BrittleStar training..."
export MUJOCO_GL=egl
export WANDB_DIR="$SCRATCH_RUNDIR"