56 lines
2.5 KiB
Markdown
56 lines
2.5 KiB
Markdown
# Checkpoint & Model Evaluation
|
|
|
|
This guide covers how to evaluate trained brittle star models, with a focus on measuring defect tolerance (amputations) across different controller architectures.
|
|
|
|
## Checkpoint Evaluation (During Training)
|
|
|
|
The `PPOTrainer` can automatically evaluate every saved checkpoint using the fast MJX backend. This is enabled via configuration.
|
|
|
|
### Configuration
|
|
|
|
In your experiment config or via CLI:
|
|
```bash
|
|
python scripts/train.py evaluation.evaluate_checkpoints=true evaluation.eval_max_steps=5000
|
|
```
|
|
|
|
Results are saved to `runs/<run_dir>/metrics/checkpoint_evaluation.csv` and synced to Weights & Biases if enabled.
|
|
|
|
## Cross-Model & Defect Tolerance Analysis
|
|
|
|
To measure how well different controllers handle damage (amputations), use `scripts/compare_models.py`. This script performs a grid search over models x morphologies.
|
|
|
|
1. Create or update a YAML file in `configs/evaluation`.
|
|
2. Run the benchmark:
|
|
|
|
```bash
|
|
python scripts/compare_models.py evaluation=poster
|
|
```
|
|
|
|
The script will evaluate every combination of model and morphology for the specified number of episodes.
|
|
|
|
The results are saved to a CSV (default: `metrics/model_comparison.csv`).
|
|
|
|
### CSV Schema
|
|
|
|
| Column | Description |
|
|
|-----------------------|--------------------------------------------------------------|
|
|
| `model_path` | Path to the trained weights. |
|
|
| `architecture` | The `morph_mode` of the model (e.g., `CENTRALIZED`, `RING`). |
|
|
| `arm_0` ... `arm_4` | Number of segments in each arm slot (0 = amputated). |
|
|
| `num_active_arms` | Total number of arms with segments > 0. |
|
|
| `seed` | The episode seed. |
|
|
| `eval_return` | Accumulated shaped reward. |
|
|
| `approx_max_velocity` | Average velocity: `(initial_dist - final_dist) / steps`. |
|
|
| `reached_target` | Whether the robot finished within the success radius. |
|
|
|
|
## Post-hoc Checkpoint Scanning
|
|
|
|
If you need to re-evaluate every saved checkpoint in a run (e.g., to generate a learning curve with different metrics):
|
|
|
|
```bash
|
|
python scripts/evaluate_checkpoints.py \
|
|
simulation.model_path=runs/<run_id>/final_model.flax \
|
|
evaluation.eval_max_steps=2000
|
|
```
|
|
|
|
This script scans the `checkpoints/` directory of the specified run and evaluates every `.flax` file it finds using the model's training morphology.
|