diff --git a/README.md b/README.md index 45bdf63..a683f16 100644 --- a/README.md +++ b/README.md @@ -22,9 +22,10 @@ uv sync --frozen ├── runs/ # Default output directory for Hydra and training artifacts ├── scripts/ # High-level entrypoints for training, simulation, and evaluation ├── src/ -│ └── brittle_star_project/ # Core library and environment logic -│ ├── evaluation/ # Checkpoint evaluation, rollout logic, and metrics persistence -│ └── trainers/ # Training loop implementations (e.g., PPO) +│ ├── brittle_star_project/ # Core library and environment logic +│ │ ├── evaluation/ # Checkpoint evaluation, rollout logic, and metrics persistence +│ │ └── trainers/ # Training loop implementations (e.g., PPO) +│ └── experiment_logger/ # Standalone logging package └── tests/ # Unit and integration tests ``` diff --git a/docs/README.md b/docs/README.md index df66859..4367711 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,6 +4,29 @@ Welcome to the Brittle Star project documentation. This codebase contains the im For the core codebase, scripts, and contribution history, visit our [GitHub Repository](https://github.com/SELab-3-2026/SEL3-2026-Groep-4). +## Core Requirements & Guides + +- **[Installation Instructions](./DEVELOPMENT.md)**: Steps to set up your development environment locally or in a devcontainer using `uv`, including GPU configuration. For High-Performance Computing (HPC) setup details, see the **[HPC Guide](./HPC.md)**. +- **[How to Run Experiments](./api/training.md)**: A complete guide on running training jobs, setting custom hyperparameters, and overriding config options using Hydra. +- **[Reproducing Experiments](./api/training.md#reproducing-experiments)**: Best practices for reproducing past training runs using exact seeds, dependencies, and automatic metadata logging. +- **[Repository Structure](#repository-structure)**: Overview of the directories and files within the codebase. + +## Repository Structure + +```text +. +├── configs/ # Hydra configuration files (YAML) +├── docs/ # Comprehensive documentation and API guides +├── runs/ # Default output directory for Hydra and training artifacts +├── scripts/ # High-level entrypoints for training, simulation, and evaluation +├── src/ +│ ├── brittle_star_project/ # Core library and environment logic +│ │ ├── evaluation/ # Checkpoint evaluation, rollout logic, and metrics persistence +│ │ └── trainers/ # Training loop implementations (e.g., PPO) +│ └── experiment_logger/ # Standalone logging package +└── tests/ # Unit and integration tests +``` + ## Design & architecture (`/design`) If you are interested in the "why did you do it like this?" diff --git a/docs/api/training.md b/docs/api/training.md index d3e6c51..e892ea2 100644 --- a/docs/api/training.md +++ b/docs/api/training.md @@ -15,6 +15,7 @@ The project uses a modular configuration system powered by [Hydra](https://hydra ``` 2. **Edit `configs/experiment/my_experiment.yaml`** to set your experiment parameters: + ```yaml # @package _global_ experiment: @@ -50,6 +51,32 @@ By default, the trainer saves checkpoints but does not evaluate them. To enable uv run python scripts/train.py evaluation.evaluate_checkpoints=true ``` +## Reproducing Experiments + +To ensure scientific validity and allow other researchers to reproduce your training runs, follow these steps: + +1. **Lock Environment Dependencies**: + Always use the exact environment lockfile when running experiments. Run: + ```bash + uv sync --frozen + ``` + This guarantees that the same package versions (including JAX, Flax, and MuJoCo) are used. + +2. **Save and Locate Configuration Metadata**: + Every time you start a training run, the configuration is fully resolved by Hydra and saved as a metadata YAML file: + - For checkpointed steps: `runs//checkpoints/_step__metadata.yaml` + - For the final model: `runs//final_model_metadata.yaml` + + This metadata file contains every active hyperparameter (e.g., learning rate, morphology configuration, PPO parameters, etc.) for that specific run. + +3. **Re-Run with Pinning**: + To reproduce a run, execute the training script with the configuration parameters specified in the metadata file, making sure to reuse the same seed: + ```bash + uv run python scripts/train.py experiment=my_experiment ppo.learning_rate=0.001 experiment.seed=42 + ``` + +--- + For more details on evaluation metrics and comparison tools, see [Evaluation](./evaluation.md). For more details on tracking your experiments, see [Tracking & Monitoring](./tracking.md).