refactor: set wandb entity and simplify READMEs
This commit is contained in:
parent
3e992a3377
commit
7e7c5bf27c
7 changed files with 21 additions and 218 deletions
|
|
@ -2,91 +2,22 @@
|
||||||
|
|
||||||
This directory contains configuration files for training experiments.
|
This directory contains configuration files for training experiments.
|
||||||
|
|
||||||
## Quick Start
|
## Usage
|
||||||
|
|
||||||
### 1. Choose a Template
|
Use `--config` with `src/train.py` to run an experiment:
|
||||||
|
|
||||||
**For Development/Testing:**
|
|
||||||
```bash
|
```bash
|
||||||
cp configs/dev_test.yaml configs/my_dev.yaml
|
python src/train.py --config configs/default_ppo.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
**For Production Training:**
|
You can overriding settings via CLI:
|
||||||
```bash
|
```bash
|
||||||
cp configs/production_training.yaml configs/my_experiment.yaml
|
python src/train.py --config configs/default_ppo.yaml --learning-rate 0.001
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. Configure Your Settings
|
## Available Configurations
|
||||||
|
|
||||||
Edit your config file and **set your wandb entity**:
|
- `default_ppo.yaml`: Baseline config.
|
||||||
```yaml
|
- `dev_test.yaml`: Fast iteration for development.
|
||||||
# ⚠️ IMPORTANT: Set this to your WandB username or team name
|
- `production_training.yaml`: Full-scale training.
|
||||||
wandb_entity: "your-wandb-username"
|
- `personal_template.yaml`: Template for team members to customize.
|
||||||
track: true # Enable WandB logging
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Run Training
|
|
||||||
|
|
||||||
**Using config file:**
|
|
||||||
```bash
|
|
||||||
python src/train.py --config configs/my_experiment.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
**Override specific parameters:**
|
|
||||||
```bash
|
|
||||||
python src/train.py --config configs/my_experiment.yaml --learning-rate 0.001 --num-envs 32
|
|
||||||
```
|
|
||||||
|
|
||||||
**Pure CLI (no config file):**
|
|
||||||
```bash
|
|
||||||
python src/train.py --track --wandb-entity your-username --total-timesteps 1000000
|
|
||||||
```
|
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
### 📊 WandB Integration
|
|
||||||
- Real-time metrics logging
|
|
||||||
- Model checkpoints as artifacts
|
|
||||||
- Run comparison and collaboration
|
|
||||||
|
|
||||||
### 🔧 Flexible Configuration
|
|
||||||
- YAML files for reproducible experiments
|
|
||||||
- CLI overrides for quick adjustments
|
|
||||||
- Team collaboration without code changes
|
|
||||||
|
|
||||||
## Configuration Templates
|
|
||||||
|
|
||||||
### `dev_test.yaml`
|
|
||||||
- Fast iteration for development
|
|
||||||
- Short runs (100K timesteps)
|
|
||||||
- Frequent checkpoints
|
|
||||||
- Small environment count
|
|
||||||
|
|
||||||
### `production_training.yaml`
|
|
||||||
- Full-scale training (50M timesteps)
|
|
||||||
- Optimized hyperparameters
|
|
||||||
- Production-ready settings
|
|
||||||
|
|
||||||
### `default_ppo.yaml`
|
|
||||||
- Baseline configuration template
|
|
||||||
- Balanced settings for most use cases
|
|
||||||
|
|
||||||
## Team Collaboration
|
|
||||||
|
|
||||||
Each team member should create their own config file:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
# configs/alice_experiment.yaml
|
|
||||||
exp_name: "alice_locomotion_v2"
|
|
||||||
track: true
|
|
||||||
wandb_project_name: "PPO-Modularity"
|
|
||||||
wandb_entity: "alice-research" # Alice's WandB username
|
|
||||||
total_timesteps: 20000000
|
|
||||||
# ... other settings
|
|
||||||
```
|
|
||||||
|
|
||||||
This allows everyone to:
|
|
||||||
- Use their own WandB account
|
|
||||||
- Run different experiments simultaneously
|
|
||||||
- Share configurations via version control
|
|
||||||
- Avoid conflicts in run names
|
|
||||||
|
|
|
||||||
|
|
@ -15,7 +15,7 @@ seed: 1
|
||||||
# Tracking settings
|
# Tracking settings
|
||||||
track: false # Set to true to enable WandB logging
|
track: false # Set to true to enable WandB logging
|
||||||
wandb_project_name: "PPO-Modularity"
|
wandb_project_name: "PPO-Modularity"
|
||||||
wandb_entity: null # Set to your WandB username or team name
|
wandb_entity: "SEL3-2026-Groep-4" # Set to your WandB username or team name
|
||||||
|
|
||||||
# Model saving
|
# Model saving
|
||||||
save_model: true
|
save_model: true
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,7 @@ seed: 123
|
||||||
# Tracking settings - IMPORTANT: Set your own wandb_entity!
|
# Tracking settings - IMPORTANT: Set your own wandb_entity!
|
||||||
track: true
|
track: true
|
||||||
wandb_project_name: "PPO-Modularity-Dev"
|
wandb_project_name: "PPO-Modularity-Dev"
|
||||||
wandb_entity: null # ⚠️ SET THIS TO YOUR WANDB USERNAME OR TEAM
|
wandb_entity: "SEL3-2026-Groep-4" # ⚠️ SET THIS TO YOUR WANDB USERNAME OR TEAM
|
||||||
|
|
||||||
# Model saving
|
# Model saving
|
||||||
save_model: true
|
save_model: true
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,7 @@ seed: 42
|
||||||
# WandB settings - ⚠️ IMPORTANT: Set your credentials!
|
# WandB settings - ⚠️ IMPORTANT: Set your credentials!
|
||||||
track: true # Enable WandB tracking
|
track: true # Enable WandB tracking
|
||||||
wandb_project_name: "PPO-Modularity"
|
wandb_project_name: "PPO-Modularity"
|
||||||
wandb_entity: "YOUR_WANDB_USERNAME" # ⚠️ CHANGE THIS to your WandB username/team
|
wandb_entity: "SEL3-2026-Groep-4" # ⚠️ CHANGE THIS to your WandB username/team
|
||||||
|
|
||||||
# Quick experiment settings (modify as needed)
|
# Quick experiment settings (modify as needed)
|
||||||
total_timesteps: 500000 # 500K for quick results
|
total_timesteps: 500000 # 500K for quick results
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,7 @@ seed: 42
|
||||||
# Tracking settings - IMPORTANT: Set your own wandb_entity!
|
# Tracking settings - IMPORTANT: Set your own wandb_entity!
|
||||||
track: true
|
track: true
|
||||||
wandb_project_name: "PPO-Modularity"
|
wandb_project_name: "PPO-Modularity"
|
||||||
wandb_entity: null # ⚠️ SET THIS TO YOUR WANDB USERNAME OR TEAM
|
wandb_entity: "SEL3-2026-Groep-4" # ⚠️ SET THIS TO YOUR WANDB USERNAME OR TEAM
|
||||||
|
|
||||||
# Model saving
|
# Model saving
|
||||||
save_model: true
|
save_model: true
|
||||||
|
|
|
||||||
|
|
@ -26,7 +26,7 @@ class PPOArgs:
|
||||||
wandb_project_name: str = "PPO-Modularity"
|
wandb_project_name: str = "PPO-Modularity"
|
||||||
|
|
||||||
# the entity (team) of wandb's project
|
# the entity (team) of wandb's project
|
||||||
wandb_entity: str | None = None
|
wandb_entity: str | None = "SEL3-2026-Groep-4"
|
||||||
|
|
||||||
# whether to capture videos of the agent performances (check out `videos` folder)
|
# whether to capture videos of the agent performances (check out `videos` folder)
|
||||||
capture_video: bool = False
|
capture_video: bool = False
|
||||||
|
|
|
||||||
|
|
@ -1,145 +1,17 @@
|
||||||
# Experiment Logger
|
# Experiment Logger
|
||||||
|
|
||||||
A lightweight, standalone logging framework for machine learning experiments with multi-backend support.
|
A lightweight logging framework supporting Weights & Biases, local JSON, and stdout.
|
||||||
|
|
||||||
## Features
|
## Usage
|
||||||
|
|
||||||
- **Multi-backend logging**: Simultaneously log to WandB, local disk (JSON), and stdout
|
|
||||||
- **Data preservation**: All metrics saved locally, even if WandB is unavailable
|
|
||||||
- **Checkpoint management**: Save model checkpoints with metadata
|
|
||||||
- **WandB integration**: Optional artifact upload for model versioning
|
|
||||||
- **Graceful degradation**: Works without WandB installed
|
|
||||||
- **Simple API**: Minimal configuration required
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
This package is included in the project. To use it in your code:
|
|
||||||
|
|
||||||
```python
|
|
||||||
from experiment_logger import UnifiedLogger
|
|
||||||
```
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from experiment_logger import UnifiedLogger
|
from experiment_logger import UnifiedLogger
|
||||||
|
|
||||||
# Initialize logger
|
logger = UnifiedLogger(run_name="my_experiment", config={"lr": 0.001})
|
||||||
logger = UnifiedLogger(
|
|
||||||
run_name="my_experiment",
|
|
||||||
config={"learning_rate": 0.001, "batch_size": 32},
|
|
||||||
project_name="MyProject",
|
|
||||||
entity="my-wandb-username", # Optional
|
|
||||||
use_wandb=True, # Set to False to disable WandB
|
|
||||||
)
|
|
||||||
|
|
||||||
# Log metrics
|
logger.log({"loss": 0.5}, step=1)
|
||||||
for step in range(100):
|
logger.save_checkpoint(params=model_params, step=1)
|
||||||
logger.log({
|
|
||||||
"loss": 1.0 / (step + 1),
|
|
||||||
"accuracy": step * 0.01,
|
|
||||||
}, step=step)
|
|
||||||
|
|
||||||
# Save checkpoint
|
|
||||||
logger.save_checkpoint(
|
|
||||||
params=model_params,
|
|
||||||
step=100,
|
|
||||||
metadata={"epoch": 1, "val_acc": 0.95},
|
|
||||||
)
|
|
||||||
|
|
||||||
# Save final model
|
|
||||||
logger.save_final_model(
|
|
||||||
params=final_params,
|
|
||||||
metadata={"final_accuracy": 0.98},
|
|
||||||
)
|
|
||||||
|
|
||||||
# Finalize (flushes remaining metrics)
|
|
||||||
logger.finish()
|
logger.finish()
|
||||||
```
|
```
|
||||||
|
|
||||||
## Context Manager
|
Logs and checkoints are saved in the `runs/` directory. If `track=True` (or `use_wandb=True`), everything is additionally synced to Weights & Biases.
|
||||||
|
|
||||||
Use as a context manager for automatic cleanup:
|
|
||||||
|
|
||||||
```python
|
|
||||||
with UnifiedLogger(run_name="my_exp", config={}) as logger:
|
|
||||||
logger.log({"metric": 1.0})
|
|
||||||
# Automatically calls finish() on exit
|
|
||||||
```
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
### Constructor Parameters
|
|
||||||
|
|
||||||
- `run_name` (str): Unique name for this run
|
|
||||||
- `config` (dict): Configuration dictionary with hyperparameters
|
|
||||||
- `project_name` (str): WandB project name (default: "PPO-Modularity")
|
|
||||||
- `entity` (str, optional): WandB entity (team/user name)
|
|
||||||
- `base_dir` (str): Base directory for local storage (default: "runs")
|
|
||||||
- `use_wandb` (bool): Enable WandB logging (default: True)
|
|
||||||
- `save_code` (bool): Save code to WandB (default: True)
|
|
||||||
|
|
||||||
### Directory Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
runs/
|
|
||||||
└── my_experiment/
|
|
||||||
├── config.json # Saved configuration
|
|
||||||
├── metrics/
|
|
||||||
│ └── metrics.jsonl # Line-delimited JSON metrics
|
|
||||||
├── checkpoints/
|
|
||||||
│ ├── checkpoint_step_100.flax
|
|
||||||
│ └── checkpoint_step_100_metadata.json
|
|
||||||
└── final_model.flax
|
|
||||||
```
|
|
||||||
|
|
||||||
## API Reference
|
|
||||||
|
|
||||||
### `log(metrics, step=None, commit=True)`
|
|
||||||
|
|
||||||
Log metrics to all backends.
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `metrics` (dict): Dictionary of metric name -> value
|
|
||||||
- `step` (int, optional): Global step counter (auto-incremented if None)
|
|
||||||
- `commit` (bool): Whether to commit to WandB immediately
|
|
||||||
|
|
||||||
### `save_checkpoint(params, step, prefix="checkpoint", metadata=None)`
|
|
||||||
|
|
||||||
Save model checkpoint to disk and optionally to WandB.
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `params`: Model parameters (Flax params or any serializable object)
|
|
||||||
- `step` (int): Current training step
|
|
||||||
- `prefix` (str): Prefix for checkpoint filename
|
|
||||||
- `metadata` (dict, optional): Additional metadata to save
|
|
||||||
|
|
||||||
### `save_final_model(params, metadata=None)`
|
|
||||||
|
|
||||||
Save the final trained model.
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `params`: Model parameters
|
|
||||||
- `metadata` (dict, optional): Metadata about the final model
|
|
||||||
|
|
||||||
### `finish()`
|
|
||||||
|
|
||||||
Finalize logging and cleanup. Flushes remaining metrics to disk.
|
|
||||||
|
|
||||||
## Usage in Projects
|
|
||||||
|
|
||||||
This logger is designed to be:
|
|
||||||
- **Project-agnostic**: Use in any ML project, not just this one
|
|
||||||
- **Framework-agnostic**: Works with JAX, PyTorch, TensorFlow, etc.
|
|
||||||
- **Minimal dependencies**: Only requires `wandb` (optional), `flax` (for serialization), and `numpy`
|
|
||||||
|
|
||||||
## Design Philosophy
|
|
||||||
|
|
||||||
1. **Never lose data**: All metrics saved locally, regardless of WandB availability
|
|
||||||
2. **Simple API**: Minimal boilerplate, easy to integrate
|
|
||||||
3. **Fail gracefully**: Missing WandB shouldn't break experiments
|
|
||||||
4. **Reproducibility**: Save full configuration with every run
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
Part of the 2026SEL3-project-BrittleStar repository.
|
|
||||||
|
|
|
||||||
Reference in a new issue