docs: experiment reproduction
This commit is contained in:
parent
f26312583a
commit
9f99470557
3 changed files with 54 additions and 3 deletions
|
|
@ -22,9 +22,10 @@ uv sync --frozen
|
|||
├── runs/ # Default output directory for Hydra and training artifacts
|
||||
├── scripts/ # High-level entrypoints for training, simulation, and evaluation
|
||||
├── src/
|
||||
│ └── brittle_star_project/ # Core library and environment logic
|
||||
│ ├── evaluation/ # Checkpoint evaluation, rollout logic, and metrics persistence
|
||||
│ └── trainers/ # Training loop implementations (e.g., PPO)
|
||||
│ ├── brittle_star_project/ # Core library and environment logic
|
||||
│ │ ├── evaluation/ # Checkpoint evaluation, rollout logic, and metrics persistence
|
||||
│ │ └── trainers/ # Training loop implementations (e.g., PPO)
|
||||
│ └── experiment_logger/ # Standalone logging package
|
||||
└── tests/ # Unit and integration tests
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -4,6 +4,29 @@ Welcome to the Brittle Star project documentation. This codebase contains the im
|
|||
|
||||
For the core codebase, scripts, and contribution history, visit our [GitHub Repository](https://github.com/SELab-3-2026/SEL3-2026-Groep-4).
|
||||
|
||||
## Core Requirements & Guides
|
||||
|
||||
- **[Installation Instructions](./DEVELOPMENT.md)**: Steps to set up your development environment locally or in a devcontainer using `uv`, including GPU configuration. For High-Performance Computing (HPC) setup details, see the **[HPC Guide](./HPC.md)**.
|
||||
- **[How to Run Experiments](./api/training.md)**: A complete guide on running training jobs, setting custom hyperparameters, and overriding config options using Hydra.
|
||||
- **[Reproducing Experiments](./api/training.md#reproducing-experiments)**: Best practices for reproducing past training runs using exact seeds, dependencies, and automatic metadata logging.
|
||||
- **[Repository Structure](#repository-structure)**: Overview of the directories and files within the codebase.
|
||||
|
||||
## Repository Structure
|
||||
|
||||
```text
|
||||
.
|
||||
├── configs/ # Hydra configuration files (YAML)
|
||||
├── docs/ # Comprehensive documentation and API guides
|
||||
├── runs/ # Default output directory for Hydra and training artifacts
|
||||
├── scripts/ # High-level entrypoints for training, simulation, and evaluation
|
||||
├── src/
|
||||
│ ├── brittle_star_project/ # Core library and environment logic
|
||||
│ │ ├── evaluation/ # Checkpoint evaluation, rollout logic, and metrics persistence
|
||||
│ │ └── trainers/ # Training loop implementations (e.g., PPO)
|
||||
│ └── experiment_logger/ # Standalone logging package
|
||||
└── tests/ # Unit and integration tests
|
||||
```
|
||||
|
||||
## Design & architecture (`/design`)
|
||||
|
||||
If you are interested in the "why did you do it like this?"
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ The project uses a modular configuration system powered by [Hydra](https://hydra
|
|||
```
|
||||
|
||||
2. **Edit `configs/experiment/my_experiment.yaml`** to set your experiment parameters:
|
||||
|
||||
```yaml
|
||||
# @package _global_
|
||||
experiment:
|
||||
|
|
@ -50,6 +51,32 @@ By default, the trainer saves checkpoints but does not evaluate them. To enable
|
|||
uv run python scripts/train.py evaluation.evaluate_checkpoints=true
|
||||
```
|
||||
|
||||
## Reproducing Experiments
|
||||
|
||||
To ensure scientific validity and allow other researchers to reproduce your training runs, follow these steps:
|
||||
|
||||
1. **Lock Environment Dependencies**:
|
||||
Always use the exact environment lockfile when running experiments. Run:
|
||||
```bash
|
||||
uv sync --frozen
|
||||
```
|
||||
This guarantees that the same package versions (including JAX, Flax, and MuJoCo) are used.
|
||||
|
||||
2. **Save and Locate Configuration Metadata**:
|
||||
Every time you start a training run, the configuration is fully resolved by Hydra and saved as a metadata YAML file:
|
||||
- For checkpointed steps: `runs/<run_dir>/checkpoints/<prefix>_step_<step>_metadata.yaml`
|
||||
- For the final model: `runs/<run_dir>/final_model_metadata.yaml`
|
||||
|
||||
This metadata file contains every active hyperparameter (e.g., learning rate, morphology configuration, PPO parameters, etc.) for that specific run.
|
||||
|
||||
3. **Re-Run with Pinning**:
|
||||
To reproduce a run, execute the training script with the configuration parameters specified in the metadata file, making sure to reuse the same seed:
|
||||
```bash
|
||||
uv run python scripts/train.py experiment=my_experiment ppo.learning_rate=0.001 experiment.seed=42
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
For more details on evaluation metrics and comparison tools, see [Evaluation](./evaluation.md).
|
||||
|
||||
For more details on tracking your experiments, see [Tracking & Monitoring](./tracking.md).
|
||||
|
|
|
|||
Reference in a new issue