fix(hpc): streamlined install

docs/HPC.md

@@ -2,125 +2,83 @@
 
 Full documentation: <https://docs.hpc.ugent.be/>
 
-## Cluster Selection
-
-Choose the appropriate cluster before submitting a job with `module swap cluster/<name>`. The default login cluster is **doduo**.
-
-> Check the current queue load at <https://shieldon.ugent.be:8083/pbsmon-web-users/>.
-
 ## Storage Overview
 
-- Run outputs are written to `$VSC_SCRATCH` during the job (fast I/O) and copied to `$VSC_DATA` at the end for persistence.
+- **Run Outputs**: Written to `$VSC_SCRATCH` during the job (fast I/O) and copied to `$VSC_DATA` at the end for persistence.
-- `$VSC_SCRATCH` may be purged periodically — do not use it as long-term storage.
+- **Virtual Environments**: Managed on **`$VSC_DATA`** by mirroring configuration files. This avoids the 3GB home quota without requiring symlinks in the project root.
-
-Check your quota: <https://account.vscentrum.be> (Usage section).
 
 ## Initial Environment Setup
 
-Run **once** after cloning the repository, but ensure you are logged into a **compute node** on the target cluster (e.g., `donphan` or `joltik`). The login node (`doduo`) will block the installation script to prevent architecture mismatches.
+Run **once** after cloning the repository. Ensure you are logged into a **compute node** on `donphan` or `joltik`. 
-
-```bash
-# 1. Swap to the target cluster
-module swap cluster/joltik
-
-# 2. Start an interactive session on a compute node
-qsub -I -l nodes=1:ppn=8:gpus=1
-
-# 3. Navigate to the project directory and run the install script
-cd "${PBS_O_WORKDIR}"
-bash scripts/hpc/install.sh
-
-# 4. Exit the interactive session once finished
-exit
-```
 
 > [!IMPORTANT]
-> Virtual environments are cluster-specific. If you want to switch from `joltik` to `accelgor`, you must re-run the `install.sh` script while logged into an interactive session on `accelgor`.
+> To avoid the **3GB home directory quota limit**, the installation script mirrors your configuration files to **`$VSC_DATA`** (25GB+ quota). The `vsc-venv` tool then automatically creates and manages the environment on the larger partition.
-
-## Interactive Debugging on donphan
-
-### Interactive shell session
 
 ```bash
-# Swap to the debug cluster (from any login node)
+# 1. Start an interactive session (donphan for debug, joltik for training)
-module swap cluster/donphan
+qsub -I -l nodes=1:ppn=8:gpus=1
 
-# Request an interactive job (1 node, 4 cores)
+# 2. Run the streamlined install script
-qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00
+cd "${PBS_O_WORKDIR}"
-
+bash scripts/hpc/install.sh
-# Once inside the job — redirect caches to scratch first
-export PIP_CACHE_DIR="$VSC_SCRATCH/.cache/pip"
-export UV_CACHE_DIR="$VSC_SCRATCH/.cache/uv"
-
-# Change to the project directory and activate environment
-cd "$PBS_O_WORKDIR"
-module load vsc-venv
-source vsc-venv --activate \
-    --modules env/hpc/modules.txt \
-    --requirements env/hpc/requirements.txt
-
-# Set headless rendering backend
-export MUJOCO_GL=egl
-
-# Run the smoke test
-python src/train.py --env-config-path configs/hpc/smoke_test.yaml
 ```
 
-### JupyterLab session (HPC web portal)
+## Interactive Debugging
 
-1. In the web portal go to **Interactive Apps → JupyterLab RHEL9**
+You can use the same `install.sh` script to quickly activate your environment for interactive work.
-2. Set the following options:
 
-   | Option              | Value                                           |
+```bash
-   |---------------------|-------------------------------------------------|
+# Request an interactive job
-   | Cluster             | `donphan (interactive/debug)`                   |
+qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00
-   | Number of nodes     | 1                                               |
-   | Number of cores     | 4                                               |
-   | JupyterLab version  | `4.2.5 GCCcore-13.3.0`                          |
-   | Custom code         | *(leave blank — vsc-venv handles modules)*      |
 
-3. Click **Launch**, wait for the session to start, then **Connect**.
+# Change to project directory and run install.sh to sync and activate
-4. In JupyterLab, select the kernel **`SEL3 (<cluster>)`**.
+cd "$PBS_O_WORKDIR"
-5. Verify GPU access:
+bash scripts/hpc/install.sh
+```
 
-   ```python
+### Verification Commands
-   import jax
+
-   print(jax.default_backend())  # expected: 'gpu'
+After installation, run these commands to ensure your environment is set up correctly:
-   print(jax.devices())          # expected: [CudaDevice(id=0)]
+
+1. **Verify Location**:
+   ```bash
+   # Confirm that NO 'venvs' folder appeared in your project root
+   ls -d venvs 2>/dev/null  # Should return 'not found'
+   
+   # Confirm the environment is on the data partition
+   python -c "import torch; print(torch.__file__)"
+   # Expected: /kyukon/data/gent/vsc... or similar
    ```
 
-> **Warning:** JAX can only be loaded by one kernel at a time. Shut down other kernels before switching notebooks.
+2. **Verify GPU Access**:
+   ```bash
+   python -c "import torch; import jax; print(f'Torch CUDA: {torch.cuda.is_available()}'); print(f'JAX Devices: {jax.devices()}')"
+   ```
+   *Expected output: `Torch CUDA: True` and `JAX Devices: [CudaDevice(id=0)]`.*
+
+3. **Verify Home Quota**:
+   ```bash
+   df -h ~ # Should show low usage (< 1GB typically)
+   ```
 
 ## Submitting Batch Training Jobs
 
 ```bash
-# Default cluster (joltik — one A100 GPU slice)
+# Submit to the default cluster (joltik)
 qsub scripts/hpc/train.pbs
 
-# Switch to a different GPU cluster first
+# To choose a different cluster (e.g. donphan debug) without touching code
-module swap cluster/accelgor
+module swap cluster/donphan && qsub scripts/hpc/train.pbs
-qsub scripts/hpc/train.pbs
 ```
 
-The job script automatically:
+The `train.pbs` script automatically handles its own activation using the mirrored configurations on `$VSC_DATA`.
-- Writes run outputs to `$VSC_SCRATCH/runs/<job_id>` during the run using the `--run_dir` argument. This ensures that frequent I/O (like tensorboard logs and checkpoints) happens on the fastest available filesystem.
-- Copies the final results to `$VSC_DATA/runs/<job_id>` on completion for long-term persistence.
-- Writes PBS stdout/stderr to `runs/brittlestar-ppo.o<job_id>` / `.e<job_id>` (standard PBS convention, relative to the project root).
-
-Monitor your jobs:
-
-```bash
-qstat          # list your jobs
-qstat -f <id>  # detailed info for a specific job
-qdel <id>      # cancel a job
-```
 
 ## Managing Dependencies
 
-`env/hpc/requirements.txt` is auto-generated by CI whenever `pyproject.toml` changes. To regenerate locally:
+`env/hpc/requirements.txt` is auto-generated from `pyproject.toml`. To regenerate:
 
 ```bash
 uv run scripts/export_hpc_requirements.py
 ```
 
-Do **not** edit `env/hpc/requirements.txt` by hand — edit `pyproject.toml` instead.
+Modules listed in `env/hpc/modules.txt` are automatically excluded from the pip requirements to save space and use HPC-optimized binaries.

env/hpc/modules.txt

@@ -1,5 +1,6 @@
 gfbf/2024a
 GCCcore/13.3.0
 Python/3.12.3-GCCcore-13.3.0
+PyTorch/2.7.1-foss-2024a-CUDA-12.6.0
 FFmpeg/7.0.2-GCCcore-13.3.0
 PyYAML/6.0.2-GCCcore-13.3.0

env/hpc/modules_gpu.txt

@@ -1 +0,0 @@
-PyTorch/2.7.1-foss-2024a-CUDA-12.6.0

pyproject.toml

@@ -23,6 +23,7 @@ dependencies = [
     "pyopengl-accelerate>=3.1.10",
     "tyro>=1.0.10",
     "wandb==0.24.2",
+    "torch>=2.4.0",
 ]
 
 [project.optional-dependencies]

scripts/hpc/install.sh

@@ -1,38 +1,31 @@
 #!/bin/bash
 # scripts/hpc/install.sh
 #
-# Usage:
+# Usage (on any compute node):
-#   module swap cluster/donphan
-#   qsub -I -l nodes=1:ppn=8:gpus=1
-#   (Wait for session to start, then:)
 #   bash scripts/hpc/install.sh
 
 set -eo pipefail
 
-# Keep caches off $VSC_HOME (quota ~3 GB).
+# Mirror configs to $VSC_DATA to avoid home quota limits (3GB)
-export PIP_CACHE_DIR="$VSC_SCRATCH/.cache/pip"
+# vsc-venv manages environments relative to the requirements file
-export UV_CACHE_DIR="$VSC_SCRATCH/.cache/uv"
+HPC_CONFIG_DIR="$VSC_DATA/2026SEL3-project/env/hpc"
-mkdir -p "$PIP_CACHE_DIR" "$UV_CACHE_DIR"
+mkdir -p "$HPC_CONFIG_DIR"
-
+cp env/hpc/*.txt "$HPC_CONFIG_DIR/"
-# Ensure we are in the project root
-if [ -n "$PBS_O_WORKDIR" ]; then
-    cd "$PBS_O_WORKDIR"
-fi
 
 module load vsc-venv
 
-echo 'Synchronizing and activating environment...'
+echo ">>> Synchronizing and activating environment (vsc-venv)..."
 source vsc-venv --activate \
-    --modules env/hpc/modules.txt \
+    --modules "$HPC_CONFIG_DIR/modules.txt" \
-    --requirements env/hpc/requirements.txt
+    --requirements "$HPC_CONFIG_DIR/requirements.txt"
 
-# Step 2: Force upgrade shared system dependencies to ensure venv precedence
+# Overlay specific NumPy/Protobuf versions to ensure venv precedence
-echo 'Applying library overlays (NumPy, Protobuf)...'
+echo ">>> Applying library overlays (NumPy, Protobuf)..."
 pip install --upgrade --no-deps numpy protobuf
 
-echo 'Installing ipykernel...'
+echo '>>> Installing ipykernel...'
-python -m ipykernel install --user --name="sel3_${VSC_INSTITUTE_CLUSTER}" \
+CLUSTER_ID="${VSC_INSTITUTE_CLUSTER:-generic}"
-    --display-name "SEL3 (${VSC_INSTITUTE_CLUSTER})"
+python -m ipykernel install --user --name="sel3_${CLUSTER_ID}" \
-
+    --display-name "SEL3 (${CLUSTER_ID})"
-echo 'Done'
 
+echo '>>> Done'

scripts/hpc/train.pbs

@@ -3,9 +3,6 @@
 #
 # Usage (from project root):
 #   qsub scripts/hpc/train.pbs
-#
-# To target a specific GPU cluster (default: joltik):
-#   module swap cluster/accelgor && qsub scripts/hpc/train.pbs
 
 #PBS -N brittlestar-ppo
 #PBS -l nodes=1:ppn=8:gpus=1
@@ -30,18 +27,23 @@ DATA_RUNDIR="$VSC_DATA/runs/$RUN_ID"
 mkdir -p "$PIP_CACHE_DIR" "$UV_CACHE_DIR" "$SCRATCH_RUNDIR" "$DATA_RUNDIR" runs/
 
 module load vsc-venv
+
+# Activate mirrored environment from $VSC_DATA to avoid home quota
+HPC_CONFIG_DIR="$VSC_DATA/2026SEL3-project/env/hpc"
 set +u
 source vsc-venv --activate \
-    --modules env/hpc/modules.txt \
+    --modules "$HPC_CONFIG_DIR/modules.txt" \
-    --requirements env/hpc/requirements.txt
+    --requirements "$HPC_CONFIG_DIR/requirements.txt"
 set -u
 
+# Ensure venv versions of NumPy/Protobuf take precedence
+pip install --upgrade --no-deps numpy protobuf > /dev/null 2>&1
+
 export MUJOCO_GL=egl
 export WANDB_DIR="$SCRATCH_RUNDIR"
 
 python src/train.py \
     --env-config-path configs/hpc/smoke_test.yaml \
     --run-dir "$SCRATCH_RUNDIR"
-    # ^^^ Replace with your production config, e.g. configs/production_training.yaml
 
 cp -r "$SCRATCH_RUNDIR/." "$DATA_RUNDIR/"

uv.lock

@@ -28,6 +28,7 @@ dependencies = [
     { name = "protobuf" },
     { name = "pyopengl" },
     { name = "pyopengl-accelerate" },
+    { name = "torch" },
     { name = "tyro" },
     { name = "wandb" },
     { name = "warp-lang" },
@@ -63,6 +64,7 @@ requires-dist = [
     { name = "protobuf", specifier = ">=5.0.0" },
     { name = "pyopengl", specifier = ">=3.1.10" },
     { name = "pyopengl-accelerate", specifier = ">=3.1.10" },
+    { name = "torch", specifier = ">=2.4.0" },
     { name = "tyro", specifier = ">=1.0.10" },
     { name = "wandb", specifier = "==0.24.2" },
     { name = "warp-lang" },