- Update train.pbs to default to smoke test for safety
- Update .gitignore and ruff.toml
- Sync uv.lock (removing python 3.11 remnants)
- Refine smoke test config
- Add run_dir and checkpoint_frequency to PPOArgs
- Update train.py to use run_dir for SummaryBoard, model saving, and loss plots
- Create configs/production_training.yaml for HPC production runs
- Update HPC.md with run_dir staging strategy details
Match comment style from previous cleanup commit:
- No section divider bars, no echo progress lines, no prose comments
on obvious commands
- Replace XDG_CACHE_HOME with explicit PIP_CACHE_DIR and UV_CACHE_DIR
- Drop MPLCONFIGDIR (matplotlib cache is negligible in size)
- Install: redirect XDG_CACHE_HOME, UV_CACHE_DIR, MPLCONFIGDIR to
$VSC_SCRATCH before any pip/module activity ($VSC_HOME is only ~3 GB)
- train.pbs:
- Set same cache env vars at job start
- Write run outputs and WandB cache to $VSC_SCRATCH during the run (fast I/O)
- Copy results to $VSC_DATA/runs/<jobid>/ on completion for persistence
- Redirect PBS stdout/stderr to $VSC_DATA/runs/<jobid>/job.{out,err}
- Add cd $PBS_O_WORKDIR as first step (HPC best practice)
- docs/HPC.md: cluster table, setup with vsc-venv, interactive debug
(shell + JupyterLab on donphan), PBS batch submission, dependency mgmt
- README.md: HPC section condensed to a single link to docs/HPC.md
- env/hpc/modules.txt: curated list of HPC EasyBuild modules loaded by vsc-venv
- env/hpc/requirements.txt: pip packages not covered by HPC modules (auto-generated)
- scripts/export_hpc_requirements.py: developer utility to regenerate requirements.txt
from pyproject.toml; to be called by CI when pyproject.toml changes
