docs: restructure docs for clarity
This commit is contained in:
parent
4fb21bbbb7
commit
511c3ebd95
6 changed files with 41 additions and 19 deletions
108
docs/api/reproduction.md
Normal file
108
docs/api/reproduction.md
Normal file
|
|
@ -0,0 +1,108 @@
|
|||
# Results & Reproduction
|
||||
|
||||
This guide explains how to access our official training logs and reproduce our results.
|
||||
|
||||
Our official training runs, model configurations, and metrics are publicly hosted on Weights & Biases (WandB).
|
||||
|
||||
---
|
||||
|
||||
## Weights & Biases (WandB) Project
|
||||
|
||||
All experiments, final models, and training logs are tracked in our public WandB project:
|
||||
|
||||
* **Official Runs Table**: [WandB final-models-v2 Table](https://wandb.ai/SEL3-2026-Groep-4/final-models-v2/table?nw=96mloffsyq)
|
||||
|
||||
This page lists the verified runs with their architecture types, morphology definitions, evaluation metrics, and final model performance.
|
||||
|
||||
### How to Reproduce a Run from WandB
|
||||
|
||||
Weights & Biases provides a built-in feature to extract the exact parameters and commands used for any given run:
|
||||
|
||||
1. Open the [WandB final-models-v2 Table](https://wandb.ai/SEL3-2026-Groep-4/final-models-v2/table?nw=96mloffsyq).
|
||||
2. Click on the name of the run you wish to reproduce to open its detail page.
|
||||
3. In the top-right corner of the run header (next to the run name, not the main workspace header), click the **three dots (`...`)** menu.
|
||||
4. Select **"Reproduce run"**. This will display the exact command-line arguments and configuration settings used to execute that run.
|
||||
|
||||
---
|
||||
|
||||
## Local & HPC Reproduction Workflow
|
||||
|
||||
To reproduce our training and evaluation phases locally or on an HPC cluster, follow the procedures below.
|
||||
|
||||
### 1. Environment Setup
|
||||
|
||||
To ensure identical package versions (including JAX, Flax, and MuJoCo), sync your environment using the lockfile:
|
||||
|
||||
```bash
|
||||
uv sync --frozen
|
||||
```
|
||||
|
||||
### 2. Training Phase
|
||||
|
||||
Run the training script using the exact parameters retrieved from WandB's "Reproduce run" page or from a downloaded `_metadata.yaml` file:
|
||||
|
||||
```bash
|
||||
uv run python scripts/train.py experiment=my_experiment ppo.learning_rate=0.001 experiment.seed=42
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Evaluation Phases
|
||||
|
||||
Reproducing our evaluation results is divided into two distinct phases:
|
||||
|
||||
### Phase 1: Determining the Best Checkpoint
|
||||
|
||||
During training, checkpoints are saved at regular intervals. To determine which of these checkpoints performed the best:
|
||||
|
||||
1. **Evaluate Checkpoints Post-Training**:
|
||||
If checkpoint evaluation was not run during training, scan the completed run's checkpoints folder by pointing to the final model path:
|
||||
|
||||
```bash
|
||||
uv run python scripts/evaluate_checkpoints.py simulation.model_path=runs/your_run_dir/final_model.flax
|
||||
```
|
||||
|
||||
This script runs deterministic rollouts for every checkpoint in `runs/your_run_dir/checkpoints/`.
|
||||
|
||||
2. **Locate the Results**:
|
||||
The evaluations are saved to:
|
||||
|
||||
```text
|
||||
runs/your_run_dir/metrics/checkpoint_evaluation.csv
|
||||
```
|
||||
|
||||
Analyze this CSV to find the checkpoint iteration with the highest average return or target success rate. This checkpoint will be used for cross-architecture comparisons.
|
||||
|
||||
### Phase 2: Comparing Checkpoints Between Architectures
|
||||
|
||||
Once the best checkpoints for each architecture are identified, they are compared under shared, standardized environments (including fault tolerance checks such as leg amputations).
|
||||
|
||||
1. **Configure the Comparison Models**:
|
||||
Open or create an evaluation config file (e.g., `configs/evaluation/poster.yaml`) and add the paths to the best checkpoints:
|
||||
|
||||
```yaml
|
||||
# configs/evaluation/poster.yaml
|
||||
evaluation:
|
||||
comparison_models:
|
||||
- runs/run_arch_centralized/checkpoints/checkpoint_best.flax
|
||||
- runs/run_arch_decentralized/checkpoints/checkpoint_best.flax
|
||||
```
|
||||
|
||||
2. **Execute the Comparison Script**:
|
||||
Run the comparison script using your config:
|
||||
|
||||
```bash
|
||||
uv run python scripts/compare_models.py evaluation=poster
|
||||
```
|
||||
|
||||
This script runs multiple sequential evaluation episodes (defined by `comparison_num_episodes` starting at `comparison_base_seed`) for every model across the selected morphologies.
|
||||
|
||||
3. **Analyze Comparison Metrics**:
|
||||
The script writes a consolidated CSV file to `metrics/model_comparison.csv` containing:
|
||||
|
||||
* **`eval_return`**: The cumulative return.
|
||||
* **`approx_max_velocity`**: The distance covered per step.
|
||||
* **`reached_target`**: Navigational success rates.
|
||||
* **`arm_0` to `arm_4`**: Active segments per arm (indicating damage/amputations).
|
||||
|
||||
This CSV can then be passed to the plotting scripts (e.g., `scripts/plots/analyze_comparisons.py`) to generate visualization plots. For details on configuration and outputs, see the **[Analysis & Plotting Guide](./analysis.md)**.
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
# Simulation & Evaluation
|
||||
# Interactive Simulation & Visualization
|
||||
|
||||
The simulation pipeline allows you to visualize trained models and evaluate their performance under various conditions.
|
||||
The simulation pipeline allows you to visualize trained models and observe their behavior under various conditions.
|
||||
|
||||
## Overview
|
||||
|
||||
|
|
@ -38,4 +38,4 @@ uv run scripts/simulate.py \
|
|||
Videos and evaluation metadata are stored in timestamped folders alongside the model:
|
||||
`runs/your_run/final_model_evaluations/eval_<timestamp>/simulation.mp4`
|
||||
|
||||
For batch evaluation and cross-model comparison, see the **[Evaluation Guide](./evaluation.md)**.
|
||||
For batch evaluation, checkpoint analysis, and cross-model architecture comparisons, see the **[Checkpoint & Model Evaluation Guide](./evaluation.md)**.
|
||||
|
|
|
|||
|
|
@ -53,10 +53,10 @@ uv run python scripts/train.py evaluation.evaluate_checkpoints=true
|
|||
|
||||
## Reproducing Experiments
|
||||
|
||||
For detailed steps on how to reproduce training runs, locate run configuration metadata, or reproduce our experiments using Weights & Biases (WandB), see the **[Results & Reproduction Guide](../reproduction.md)**.
|
||||
For detailed steps on how to reproduce training runs, locate run configuration metadata, or reproduce our experiments using Weights & Biases (WandB), see the **[Results & Reproduction Guide](./reproduction.md)**.
|
||||
|
||||
---
|
||||
|
||||
For more details on evaluation metrics and comparison tools, see [Evaluation](./evaluation.md).
|
||||
For more details on evaluation metrics and comparison tools, see [Checkpoint & Model Evaluation](./evaluation.md).
|
||||
|
||||
For more details on tracking your experiments, see [Tracking & Monitoring](./tracking.md).
|
||||
|
|
|
|||
Reference in a new issue