80 lines
2.3 KiB
Markdown
80 lines
2.3 KiB
Markdown
# HPC Guide
|
|
|
|
Full documentation: <https://docs.hpc.ugent.be/>
|
|
|
|
## Storage Overview
|
|
|
|
- **Run Outputs**: Written to `$VSC_SCRATCH` during the job (fast I/O) and copied to `$VSC_DATA` at the end for persistence.
|
|
- **Virtual Environments**: Managed on **`$VSC_DATA`** by mirroring configuration files. This avoids the 3GB home quota without requiring symlinks in the project root.
|
|
|
|
## Initial Environment Setup
|
|
|
|
Run **once** after cloning the repository. This script handles all modules, mirroring, and environment synchronization.
|
|
|
|
```bash
|
|
# Option A: Interactive (on a compute node)
|
|
module swap cluster/donphan # or joltik
|
|
qsub -I -l nodes=1:gpus=1
|
|
cd "${PBS_O_WORKDIR}"
|
|
bash scripts/hpc/install.sh
|
|
|
|
# Option B: Batch (Run in background)
|
|
qsub scripts/hpc/install.sh
|
|
```
|
|
|
|
## Interactive Debugging
|
|
|
|
To activate your environment for interactive work, simply run the same `install.sh` script.
|
|
|
|
```bash
|
|
qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00
|
|
cd "$PBS_O_WORKDIR"
|
|
bash scripts/hpc/install.sh
|
|
```
|
|
|
|
### Verification Commands
|
|
|
|
After installation, run these commands to ensure your environment is set up correctly:
|
|
|
|
1. **Verify Quota Safety**:
|
|
```bash
|
|
ls -d venvs 2>/dev/null && echo "FAIL" || echo ">>> PASS: Project root is clean."
|
|
```
|
|
2. **Verify Library Versions (NumPy Fix)**:
|
|
```bash
|
|
python -c "import numpy; print(f'NumPy: {numpy.__version__}')"
|
|
# Expected: 2.x.x (Venv version), not 1.2x (System version)
|
|
```
|
|
3. **Verify GPU Access**:
|
|
```bash
|
|
python -c "import torch, jax; print(f'GPU: {torch.cuda.is_available()}'); print(f'JAX: {jax.devices()}')"
|
|
```
|
|
|
|
## PR Verification (Quick Start)
|
|
|
|
If you are a reviewer verifying a PR, run this single block:
|
|
|
|
```bash
|
|
git checkout <pr-branch>
|
|
module swap cluster/donphan
|
|
qsub -I -l nodes=1:gpus=1
|
|
|
|
# Inside the interactive session:
|
|
cd "$PBS_O_WORKDIR"
|
|
bash scripts/hpc/install.sh
|
|
python -c "import torch, jax; print(torch.cuda.is_available()); print(jax.devices())"
|
|
exit
|
|
|
|
# Verify batch submission
|
|
qsub scripts/hpc/train.pbs
|
|
```
|
|
|
|
## Managing Dependencies
|
|
|
|
`env/hpc/requirements.txt` is auto-generated from `pyproject.toml`. To regenerate:
|
|
|
|
```bash
|
|
uv run scripts/export_hpc_requirements.py
|
|
```
|
|
|
|
Modules listed in `env/hpc/modules.txt` are automatically excluded from the pip requirements to save space and use HPC-optimized binaries.
|