From 4fb21bbbb77d35a15d3305107e5b989bf12bf139 Mon Sep 17 00:00:00 2001 From: Tibo De Peuter Date: Wed, 20 May 2026 14:42:13 +0200 Subject: [PATCH] docs: reproduce results --- README.md | 4 ++ docs/README.md | 2 +- docs/api/training.md | 22 +-------- docs/reproduction.md | 108 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 114 insertions(+), 22 deletions(-) create mode 100644 docs/reproduction.md diff --git a/README.md b/README.md index a683f16..9bb9ee2 100644 --- a/README.md +++ b/README.md @@ -49,6 +49,10 @@ For detailed instructions on how to use the project, please refer to the **[API 4. **Compare fault tolerance of models:** See [Checkpoint & Model Evaluation](docs/api/evaluation.md) +## Results & Reproduction + +See **[docs/reproduction.md](docs/reproduction.md)** to learn how to access our public [Weights & Biases (WandB) project](https://wandb.ai/SEL3-2026-Groep-4/final-models-v2?nw=96mloffsyq), retrieve specific run parameters, and run the training/evaluation reproduction workflow. + ## HPC See **[docs/HPC.md](docs/HPC.md)** for the full guide, including environment setup, cluster selection, interactive debugging, and job submission. diff --git a/docs/README.md b/docs/README.md index 4367711..1e0c21d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,7 +8,7 @@ For the core codebase, scripts, and contribution history, visit our [GitHub Repo - **[Installation Instructions](./DEVELOPMENT.md)**: Steps to set up your development environment locally or in a devcontainer using `uv`, including GPU configuration. For High-Performance Computing (HPC) setup details, see the **[HPC Guide](./HPC.md)**. - **[How to Run Experiments](./api/training.md)**: A complete guide on running training jobs, setting custom hyperparameters, and overriding config options using Hydra. -- **[Reproducing Experiments](./api/training.md#reproducing-experiments)**: Best practices for reproducing past training runs using exact seeds, dependencies, and automatic metadata logging. +- **[Results & Reproduction](./reproduction.md)**: Guide on how to access our public WandB training runs table and reproduce our training and evaluation phases (determining the best checkpoint vs. comparing architectures). - **[Repository Structure](#repository-structure)**: Overview of the directories and files within the codebase. ## Repository Structure diff --git a/docs/api/training.md b/docs/api/training.md index e892ea2..a3b813a 100644 --- a/docs/api/training.md +++ b/docs/api/training.md @@ -53,27 +53,7 @@ uv run python scripts/train.py evaluation.evaluate_checkpoints=true ## Reproducing Experiments -To ensure scientific validity and allow other researchers to reproduce your training runs, follow these steps: - -1. **Lock Environment Dependencies**: - Always use the exact environment lockfile when running experiments. Run: - ```bash - uv sync --frozen - ``` - This guarantees that the same package versions (including JAX, Flax, and MuJoCo) are used. - -2. **Save and Locate Configuration Metadata**: - Every time you start a training run, the configuration is fully resolved by Hydra and saved as a metadata YAML file: - - For checkpointed steps: `runs//checkpoints/_step__metadata.yaml` - - For the final model: `runs//final_model_metadata.yaml` - - This metadata file contains every active hyperparameter (e.g., learning rate, morphology configuration, PPO parameters, etc.) for that specific run. - -3. **Re-Run with Pinning**: - To reproduce a run, execute the training script with the configuration parameters specified in the metadata file, making sure to reuse the same seed: - ```bash - uv run python scripts/train.py experiment=my_experiment ppo.learning_rate=0.001 experiment.seed=42 - ``` +For detailed steps on how to reproduce training runs, locate run configuration metadata, or reproduce our experiments using Weights & Biases (WandB), see the **[Results & Reproduction Guide](../reproduction.md)**. --- diff --git a/docs/reproduction.md b/docs/reproduction.md new file mode 100644 index 0000000..f6fba01 --- /dev/null +++ b/docs/reproduction.md @@ -0,0 +1,108 @@ +# Results & Reproduction + +This guide explains how to access our official training logs and reproduce our results. + +Our official training runs, model configurations, and metrics are publicly hosted on Weights & Biases (WandB). + +--- + +## Weights & Biases (WandB) Project + +All experiments, final models, and training logs are tracked in our public WandB project: + +* **Official Runs Table**: [WandB final-models-v2 Table](https://wandb.ai/SEL3-2026-Groep-4/final-models-v2/table?nw=96mloffsyq) + +This page lists the verified runs with their architecture types, morphology definitions, evaluation metrics, and final model performance. + +### How to Reproduce a Run from WandB + +Weights & Biases provides a built-in feature to extract the exact parameters and commands used for any given run: + +1. Open the [WandB final-models-v2 Table](https://wandb.ai/SEL3-2026-Groep-4/final-models-v2/table?nw=96mloffsyq). +2. Click on the name of the run you wish to reproduce to open its detail page. +3. In the top-right corner of the run header (next to the run name, not the main workspace header), click the **three dots (`...`)** menu. +4. Select **"Reproduce run"**. This will display the exact command-line arguments and configuration settings used to execute that run. + +--- + +## Local & HPC Reproduction Workflow + +To reproduce our training and evaluation phases locally or on an HPC cluster, follow the procedures below. + +### 1. Environment Setup + +To ensure identical package versions (including JAX, Flax, and MuJoCo), sync your environment using the lockfile: + +```bash +uv sync --frozen +``` + +### 2. Training Phase + +Run the training script using the exact parameters retrieved from WandB's "Reproduce run" page or from a downloaded `_metadata.yaml` file: + +```bash +uv run python scripts/train.py experiment=my_experiment ppo.learning_rate=0.001 experiment.seed=42 +``` + +--- + +## Evaluation Phases + +Reproducing our evaluation results is divided into two distinct phases: + +### Phase 1: Determining the Best Checkpoint + +During training, checkpoints are saved at regular intervals. To determine which of these checkpoints performed the best: + +1. **Evaluate Checkpoints Post-Training**: + If checkpoint evaluation was not run during training, scan the completed run's checkpoints folder by pointing to the final model path: + + ```bash + uv run python scripts/evaluate_checkpoints.py simulation.model_path=runs/your_run_dir/final_model.flax + ``` + + This script runs deterministic rollouts for every checkpoint in `runs/your_run_dir/checkpoints/`. + +2. **Locate the Results**: + The evaluations are saved to: + + ```text + runs/your_run_dir/metrics/checkpoint_evaluation.csv + ``` + + Analyze this CSV to find the checkpoint iteration with the highest average return or target success rate. This checkpoint will be used for cross-architecture comparisons. + +### Phase 2: Comparing Checkpoints Between Architectures + +Once the best checkpoints for each architecture are identified, they are compared under shared, standardized environments (including fault tolerance checks such as leg amputations). + +1. **Configure the Comparison Models**: + Open or create an evaluation config file (e.g., `configs/evaluation/poster.yaml`) and add the paths to the best checkpoints: + + ```yaml + # configs/evaluation/poster.yaml + evaluation: + comparison_models: + - runs/run_arch_centralized/checkpoints/checkpoint_best.flax + - runs/run_arch_decentralized/checkpoints/checkpoint_best.flax + ``` + +2. **Execute the Comparison Script**: + Run the comparison script using your config: + + ```bash + uv run python scripts/compare_models.py evaluation=poster + ``` + + This script runs multiple sequential evaluation episodes (defined by `comparison_num_episodes` starting at `comparison_base_seed`) for every model across the selected morphologies. + +3. **Analyze Comparison Metrics**: + The script writes a consolidated CSV file to `metrics/model_comparison.csv` containing: + + * **`eval_return`**: The cumulative return. + * **`approx_max_velocity`**: The distance covered per step. + * **`reached_target`**: Navigational success rates. + * **`arm_0` to `arm_4`**: Active segments per arm (indicating damage/amputations). + +This CSV can then be passed to the plotting scripts (e.g., `scripts/plots/analyze_comparisons.py`) to generate visualization plots. For details on configuration and outputs, see the **[Analysis & Plotting Guide](./api/analysis.md)**.