1
Fork 0

docs: restructure docs for clarity

This commit is contained in:
Tibo De Peuter 2026-05-20 14:54:01 +02:00
parent 4fb21bbbb7
commit 511c3ebd95
6 changed files with 41 additions and 19 deletions

108
docs/api/reproduction.md Normal file
View file

@ -0,0 +1,108 @@
# Results & Reproduction
This guide explains how to access our official training logs and reproduce our results.
Our official training runs, model configurations, and metrics are publicly hosted on Weights & Biases (WandB).
---
## Weights & Biases (WandB) Project
All experiments, final models, and training logs are tracked in our public WandB project:
* **Official Runs Table**: [WandB final-models-v2 Table](https://wandb.ai/SEL3-2026-Groep-4/final-models-v2/table?nw=96mloffsyq)
This page lists the verified runs with their architecture types, morphology definitions, evaluation metrics, and final model performance.
### How to Reproduce a Run from WandB
Weights & Biases provides a built-in feature to extract the exact parameters and commands used for any given run:
1. Open the [WandB final-models-v2 Table](https://wandb.ai/SEL3-2026-Groep-4/final-models-v2/table?nw=96mloffsyq).
2. Click on the name of the run you wish to reproduce to open its detail page.
3. In the top-right corner of the run header (next to the run name, not the main workspace header), click the **three dots (`...`)** menu.
4. Select **"Reproduce run"**. This will display the exact command-line arguments and configuration settings used to execute that run.
---
## Local & HPC Reproduction Workflow
To reproduce our training and evaluation phases locally or on an HPC cluster, follow the procedures below.
### 1. Environment Setup
To ensure identical package versions (including JAX, Flax, and MuJoCo), sync your environment using the lockfile:
```bash
uv sync --frozen
```
### 2. Training Phase
Run the training script using the exact parameters retrieved from WandB's "Reproduce run" page or from a downloaded `_metadata.yaml` file:
```bash
uv run python scripts/train.py experiment=my_experiment ppo.learning_rate=0.001 experiment.seed=42
```
---
## Evaluation Phases
Reproducing our evaluation results is divided into two distinct phases:
### Phase 1: Determining the Best Checkpoint
During training, checkpoints are saved at regular intervals. To determine which of these checkpoints performed the best:
1. **Evaluate Checkpoints Post-Training**:
If checkpoint evaluation was not run during training, scan the completed run's checkpoints folder by pointing to the final model path:
```bash
uv run python scripts/evaluate_checkpoints.py simulation.model_path=runs/your_run_dir/final_model.flax
```
This script runs deterministic rollouts for every checkpoint in `runs/your_run_dir/checkpoints/`.
2. **Locate the Results**:
The evaluations are saved to:
```text
runs/your_run_dir/metrics/checkpoint_evaluation.csv
```
Analyze this CSV to find the checkpoint iteration with the highest average return or target success rate. This checkpoint will be used for cross-architecture comparisons.
### Phase 2: Comparing Checkpoints Between Architectures
Once the best checkpoints for each architecture are identified, they are compared under shared, standardized environments (including fault tolerance checks such as leg amputations).
1. **Configure the Comparison Models**:
Open or create an evaluation config file (e.g., `configs/evaluation/poster.yaml`) and add the paths to the best checkpoints:
```yaml
# configs/evaluation/poster.yaml
evaluation:
comparison_models:
- runs/run_arch_centralized/checkpoints/checkpoint_best.flax
- runs/run_arch_decentralized/checkpoints/checkpoint_best.flax
```
2. **Execute the Comparison Script**:
Run the comparison script using your config:
```bash
uv run python scripts/compare_models.py evaluation=poster
```
This script runs multiple sequential evaluation episodes (defined by `comparison_num_episodes` starting at `comparison_base_seed`) for every model across the selected morphologies.
3. **Analyze Comparison Metrics**:
The script writes a consolidated CSV file to `metrics/model_comparison.csv` containing:
* **`eval_return`**: The cumulative return.
* **`approx_max_velocity`**: The distance covered per step.
* **`reached_target`**: Navigational success rates.
* **`arm_0` to `arm_4`**: Active segments per arm (indicating damage/amputations).
This CSV can then be passed to the plotting scripts (e.g., `scripts/plots/analyze_comparisons.py`) to generate visualization plots. For details on configuration and outputs, see the **[Analysis & Plotting Guide](./analysis.md)**.

View file

@ -1,6 +1,6 @@
# Simulation & Evaluation
# Interactive Simulation & Visualization
The simulation pipeline allows you to visualize trained models and evaluate their performance under various conditions.
The simulation pipeline allows you to visualize trained models and observe their behavior under various conditions.
## Overview
@ -38,4 +38,4 @@ uv run scripts/simulate.py \
Videos and evaluation metadata are stored in timestamped folders alongside the model:
`runs/your_run/final_model_evaluations/eval_<timestamp>/simulation.mp4`
For batch evaluation and cross-model comparison, see the **[Evaluation Guide](./evaluation.md)**.
For batch evaluation, checkpoint analysis, and cross-model architecture comparisons, see the **[Checkpoint & Model Evaluation Guide](./evaluation.md)**.

View file

@ -53,10 +53,10 @@ uv run python scripts/train.py evaluation.evaluate_checkpoints=true
## Reproducing Experiments
For detailed steps on how to reproduce training runs, locate run configuration metadata, or reproduce our experiments using Weights & Biases (WandB), see the **[Results & Reproduction Guide](../reproduction.md)**.
For detailed steps on how to reproduce training runs, locate run configuration metadata, or reproduce our experiments using Weights & Biases (WandB), see the **[Results & Reproduction Guide](./reproduction.md)**.
---
For more details on evaluation metrics and comparison tools, see [Evaluation](./evaluation.md).
For more details on evaluation metrics and comparison tools, see [Checkpoint & Model Evaluation](./evaluation.md).
For more details on tracking your experiments, see [Tracking & Monitoring](./tracking.md).