CLI Arguments: Supports the same --show_titles, --font_size, and --fig_width/height flags as the comparison script.
Outputs
-Generates three plots (PNG & SVG):
-1. convergence_comparison: Grouped horizontal bar chart.
-2. progress_reward_curves: Line plots of reward over time.
-3. progress_velocity_curves: Line plots of velocity over time.
+Generates three plots (PNG & SVG):
+
+convergence_comparison: Grouped horizontal bar chart.progress_reward_curves: Line plots of reward over time.progress_velocity_curves: Line plots of velocity over time.
Poster Integration (Figma)
SVG & Scaling
-We recommend using the SVG outputs for poster design in Figma:
-1. No Resolution Loss: SVGs are vector-based and will remain sharp at any size.
-2. Native Text: Text in the SVG imports as native text layers in Figma.
-3. Exact Font Matching: To ensure a 28pt font in the plot matches a 28pt font in your poster, set the --fig_width and --fig_height to match the physical dimensions of the plot box in your Figma layout.
-4. Editable: You can "Ungroup" the SVG in Figma to manually move labels, adjust colors, or tweak individual bars.
+We recommend using the SVG outputs for poster design in Figma:
+
+- No Resolution Loss: SVGs are vector-based and will remain sharp at any size.
+- Native Text: Text in the SVG imports as native text layers in Figma.
+- Exact Font Matching: To ensure a
28pt font in the plot matches a 28pt font in your poster, set the --fig_width and --fig_height to match the physical dimensions of the plot box in your Figma layout.
+- Editable: You can "Ungroup" the SVG in Figma to manually move labels, adjust colors, or tweak individual bars.
+
Image Placeholders
The comparison charts include light-gray square placeholders below the X-axis. These are designed as guides; in Figma, you can drop your morphology renders or illustrations directly on top of these squares.
diff --git a/api/environment/index.html b/api/environment/index.html
index c101dfa..8082750 100644
--- a/api/environment/index.html
+++ b/api/environment/index.html
@@ -1179,20 +1179,26 @@
that creates instances of the environment/morphologies/... It uses the
configuration classes defined in env_config.py to create the instances.
Configuration
-The data classes in env_config have default values as stated in the tutorials.
-* MorphologyConfig: configuration for the morphology of the brittle star. Contains
-number of arms, number of segments per arm, and control mode.
-* ArenaConfig: configuration for the arena. Sets the size of the arena, whether to
-set the ground floor to sand, attach a target and sizes of the walls.
-* EnvConfig: configuration for the environment. These set shared settings
-such as camera locations, simulation time and the task.
+The data classes in env_config have default values as stated in the tutorials.
+
+- MorphologyConfig: configuration for the morphology of the brittle star. Contains
+number of arms, number of segments per arm, and control mode.
+- ArenaConfig: configuration for the arena. Sets the size of the arena, whether to
+set the ground floor to sand, attach a target and sizes of the walls.
+- EnvConfig: configuration for the environment. These set shared settings
+such as camera locations, simulation time and the task.
+
Backend and Task enums
-The Backend enum specifies either an MJC or MJX backend.
-* MJC: runs on CPU
-* MJX: uses jax on the gpu
-The Task enum specifies which task to use. 2 items are present:
-* DIRECTED_LOCOMOTION: move to a target location
-* LIGHT_ESCAPE: situation where the robot must move to a darker location
+The Backend enum specifies either an MJC or MJX backend.
+
+- MJC: runs on CPU
+- MJX: uses jax on the gpu
+
+The Task enum specifies which task to use. 2 items are present:
+
+- DIRECTED_LOCOMOTION: move to a target location
+- LIGHT_ESCAPE: situation where the robot must move to a darker location
+
diff --git a/design/reward_function/index.html b/design/reward_function/index.html
index 2e0305c..c31be9e 100644
--- a/design/reward_function/index.html
+++ b/design/reward_function/index.html
@@ -1182,8 +1182,7 @@ inputs must be distributed fairly to guarantee an objective comparison between d
The reward function is centered around minimizing the distance to the goal or maximizing the movement towards the goal
within a finite number of timesteps $T$.
To motivate efficient movement, the amount of timesteps taken to reach the goal will be used as penalty.
-An extra penalty based on movement relative to the current step and
-the previous is used to penalize a movement away from the target.
+An extra penalty based on movement relative to the current step and the previous is used to penalize a movement away from the target.
From reward to PPO
The resulting reward is passed to our PPO library. Our critic network (value function) predicts how good our eventual reward will be for the current state, this value is combined with the reward from the reward function to get advantages. These advantages are then used to calculate the losses to update both our critic and actor pipeline.
diff --git a/search/search_index.json b/search/search_index.json
index 6fba1c8..8c9313a 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"],"fields":{"title":{"boost":1000.0},"text":{"boost":1.0},"tags":{"boost":1000000.0}}},"docs":[{"location":"","title":"Documentation","text":""},{"location":"#design-architecture-design","title":"Design & architecture (/design)","text":"If you are interested in the \"why did you do it like this?\"
- Actor/critic architecture: Description of the actor-critic pipeline.
- Communication: Message propagation, Nerve-Net style.
- Controllers: Macroscopig brain toplogy, centralized, arm-level, segment-level.
- Input/output: Description of the model's input and output.
- Learning algorithm: RL techniques, i.e. PPO.
- Reward function: Goals, fitness tracking, and reward structures.
"},{"location":"#api-reference-api","title":"API reference (/api)","text":"If you are interested in the \"how do I use it?\"
- Training: How to configure and run experiments.
- Tracking & Monitoring: Setting up WandB and TensorBoard to monitor runs.
- Simulation: Visualizing and evaluating models.
- Environment: MuJoCo environment interaction and configuration.
- Analysis: Comparing checkpoints and generating plots.
- Evaluation: Evaluating checkpoints and comparing fault tolerance.
"},{"location":"CONTRIBUTING/","title":"Contribution Guidelines","text":"This document outlines the contribution protocols for the scientific software engineering project focusing on bio-inspired control architectures for brittle-star-like robots. The primary objective of this project is to produce scientific insight, rather than a commercial product.
"},{"location":"CONTRIBUTING/#1-scientific-context-methodology","title":"1. Scientific Context & Methodology","text":" - Research Focus: The goal is to study how controller modularity affects learning speed, coordination, and fault tolerance in brittle-star locomotion.
- Hypothesis-Driven Design: Clear hypotheses must dictate a structured methodology and rigorous evaluation. All design decisions must be formally documented prior to implementation.
- Scaffolding Approach: Development must start with simple setups before progressively increasing the complexity of environments and morphologies.
- Evaluation of Results: Negative results possess scientific validity when thoroughly analyzed. If a controller fails to learn locomotion, providing a comprehensive analysis of the failure is considered a strong scientific contribution.
- Reproducibility: Contributors must utilize fixed library versions. Configuration systems (such as json, gin, or yaml) must be employed to ensure reproducible runs.
"},{"location":"CONTRIBUTING/#2-clean-code-code-quality","title":"2. Clean Code & Code Quality","text":"Code readability is paramount, as code is read far more frequently than it is written.
- Naming Conventions: Variables and functions must utilize consistent, intention-revealing names. A long, descriptive name is strictly preferred over a short name accompanied by a comment.
- Function Design: Functions must be modular and adhere to the single responsibility principle. Arguments must be minimized, and boolean flag arguments controlling behavior should be avoided.
- Commenting: Code must document the \"how,\" while comments are strictly reserved for documenting the \"why\". Commented-out code is prohibited and must be deleted via version control.
- YAGNI: Contributors must adhere to the \"You Aren't Gonna Need It\" (YAGNI) principle and actively avoid premature optimization.
- Notebooks: Jupyter Notebooks are strictly limited to quick prototyping, tutorials, demonstrations, or post-processing analysis. They are explicitly forbidden for general software development because they discourage modularity.
"},{"location":"CONTRIBUTING/#3-version-control-repository-structure","title":"3. Version Control & Repository Structure","text":" - Git Practices: Commits must be frequent and small. Each commit should relate to exactly one piece of functionality.
- Branching Strategy: The
dev branch serves as the integration branch for pushing and merging code. Only stable releases may be pushed to the main branch. - Artifact Management: Data files, trained models, and large datasets must never be committed directly to Git. Git Large File Storage (LFS) must be used for tracking large files. All developers must have
git-lfs installed locally (see DEVELOPMENT.md for setup). - Repository Layout: The repository must maintain the following core directories:
src/ for algorithms, env/ for MuJoCo wrappers, config/ for experiment configurations, experiments/ for scripts, docs/ for Doxygen or ReadTheDocs documentation, and tests/ for unit tests.
"},{"location":"CONTRIBUTING/#4-architecture-tooling","title":"4. Architecture & Tooling","text":" - Algorithms & Frameworks: Proximal Policy Optimization (PPO) is the recommended baseline algorithm. CleanRL should be used as a starting point and adapted for continuous action spaces. All Artificial Neural Network (ANN) controller architectures must be implemented using Flax.
- Simulation: The simulation environment utilizes a MuJoCo brittle star. XML MuJoCo structures must remain realistic and respect morphological constraints.
- Experiment Tracking: Weights & Biases (wandb) must be utilized for tracking and logging all experiments.
- Code Styling: All code must conform to the chosen style guide (Google standard). This is enforced via
uv using ruff and pre-commit hooks.
"},{"location":"CONTRIBUTING/#5-ai-assisted-development-code-review","title":"5. AI-Assisted Development & Code Review","text":"This project supports AI-assisted development to enhance productivity, but contributors must take full responsibility for all AI-generated outputs.
- Self-Review Requirement: Contributors must thoroughly self-review all AI-assisted code, documentation, and configurations before requesting peer review. This includes verifying correctness, adherence to project standards, scientific validity, and integration with existing code.
- Quality Standards: AI-generated content must meet the same rigorous standards as manually written code, including proper testing, documentation, and alignment with the scientific methodology outlined in Section 1.
- Available Skills: This project provides specific AI skills for common tasks (located in
.agents/skills/), including linting and testing workflows. Contributors should leverage these skills to maintain consistency and quality. - Transparency: When using AI assistance for complex algorithmic decisions or scientific design choices, contributors should document the rationale in commit messages or code comments where appropriate.
"},{"location":"DEVELOPMENT/","title":"Development Guide","text":"This guide outlines how to set up the development environment for this project, prioritizing reproducible builds, environment parity, and cross-hardware compatibility.
"},{"location":"DEVELOPMENT/#reproducibility-uv","title":"Reproducibility &uv","text":"This project uses uv to manage dependencies and virtual environments. The uv.lock file is the absolute source of truth for package versions and must always be committed.
"},{"location":"DEVELOPMENT/#source-of-truth","title":"Source of Truth","text":" - Never modify
uv.lock manually. - To add a dependency, run
uv add <package>. - To update dependencies, run
uv lock --upgrade. - To sync your environment with the lockfile, run
uv sync --frozen.
"},{"location":"DEVELOPMENT/#git-lfs-critical","title":"Git LFS (Critical)","text":"All developers must have Git LFS installed locally. This repository tracks model weights (.pt, .safetensors, etc.), recordings (.mp4), and datasets using Git LFS.
- Setup: Run
git lfs install after cloning this repository. If you are using the .devcontainer or flake.nix, LFS is typically available automatically. - If you clone without LFS installed, run
git lfs pull after installation to fetch the actual data files instead of the small pointer files.
"},{"location":"DEVELOPMENT/#devcontainer-setup-recommended","title":"Devcontainer Setup (Recommended)","text":"The devcontainer provides an identical experience to local development but with all system dependencies pre-configured. It automatically detects your hardware (GPU vs CPU) and syncs the appropriate dependencies.
"},{"location":"DEVELOPMENT/#prerequisites","title":"Prerequisites","text":" - Docker Desktop or Docker Engine.
- NVIDIA Container Toolkit (for GPU support).
"},{"location":"DEVELOPMENT/#setup-for-vs-code","title":"Setup for VS Code","text":" - Install the Dev Containers extension.
- Open the project and click Reopen in Container.
- On first launch, the
post-create.sh script will: - Detect if an NVIDIA GPU is available via
nvidia-smi. - Run
uv sync --frozen --extra cuda if a GPU is found. - Run
uv sync --frozen otherwise. - The environment is stored in a named volume for
.venv to ensure persistence and performance.
"},{"location":"DEVELOPMENT/#setup-for-jetbrains-ides","title":"Setup for JetBrains IDEs","text":" - The IDE will detect the
.devcontainer/devcontainer.json file. - The environment is pre-configured to point to
/workspaces/project/.venv. - The hardware-aware sync will run automatically during container creation.
"},{"location":"DEVELOPMENT/#local-development-alternative","title":"Local Development (Alternative)","text":"If you prefer not to use Docker: 1. Install uv. 2. Run uv sync --frozen (CPU) or uv sync --frozen --extra cuda (GPU).
"},{"location":"DEVELOPMENT/#hardware-acceleration-jax","title":"Hardware Acceleration (JAX)","text":"Verify your setup by running the JAX initialization test:
uv run pytest tests/test_jax_init.py\n
In the devcontainer, this will succeed on both CPU and GPU. A GpuDevice is expected if a GPU is detected and the cuda extra was installed."},{"location":"DEVELOPMENT/#logging-monitoring","title":"Logging & Monitoring","text":"This project uses a unified logging system through the experiment_logger package.
- Usage in Code: To use the logger in your scripts, refer to the package README for the API reference.
- WandB/TensorBoard Setup: For information on how to configure tracking for experiments, see the Tracking & Monitoring API Guide.
The logger automatically detects if it is running in an interactive terminal or a non-interactive environment (like an HPC Slurm job), adjusting progress bars and fallback modes accordingly.
"},{"location":"HPC/","title":"HPC Guide","text":"Full documentation: https://docs.hpc.ugent.be/
"},{"location":"HPC/#storage-overview","title":"Storage Overview","text":" - Run Outputs: Written to
$VSC_SCRATCH during the job (fast I/O) and copied to $VSC_DATA at the end for persistence. - Virtual Environments: Managed on
$VSC_DATA by mirroring configuration files. This avoids the 3GB home quota without requiring symlinks in the project root.
"},{"location":"HPC/#initial-environment-setup","title":"Initial Environment Setup","text":"Run once after cloning the repository. This script handles all modules, mirroring, and environment synchronization.
# Option A: Interactive (on a compute node)\nmodule swap cluster/donphan # Debug cluster (CPU only)\n# OR for GPU clusters:\n# module swap cluster/joltik\n# module swap cluster/accelgor\n# module swap cluster/litleo\n\nqsub -I -l nodes=1:gpus=1 # Only for GPU clusters\ncd \"${PBS_O_WORKDIR}\"\nbash scripts/hpc/install.sh\n\n# Option B: Batch (Run in background)\n# NOTE: GPU clusters (joltik/accelgor/litleo) require -l gpus=1 at runtime\nqsub -l gpus=1 scripts/hpc/install.sh\n
"},{"location":"HPC/#production-vs-debug-clusters","title":"Production vs. Debug Clusters","text":"Our scripts are cluster-agnostic and do not have hardcoded GPU requirements. Instead, you must request GPUs at runtime using the -l gpus=1 flag when submitting to a production GPU cluster.
"},{"location":"HPC/#debugging-donphan","title":"Debugging (Donphan)","text":"The donphan cluster does not support GPUs. Simply run the scripts without extra resource flags:
module swap cluster/donphan\nqsub scripts/hpc/train.pbs\n
"},{"location":"HPC/#production-joltik-accelgor-litleo","title":"Production (Joltik, Accelgor, Litleo)","text":"These clusters provide GPU acceleration and require a GPU request at runtime:
module swap cluster/joltik # or accelgor/litleo\nqsub -l gpus=1 scripts/hpc/train.pbs\n
"},{"location":"HPC/#interactive-debugging","title":"Interactive Debugging","text":"To activate your environment for interactive work, simply run the same install.sh script.
qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00\ncd \"$PBS_O_WORKDIR\"\nbash scripts/hpc/install.sh\n
"},{"location":"HPC/#verification-commands","title":"Verification Commands","text":"After installation, run these commands to ensure your environment is set up correctly:
- Verify Quota Safety:
ls -d venvs 2>/dev/null && echo \"FAIL\" || echo \">>> PASS: Project root is clean.\"\n
- Verify Library Versions (NumPy Fix):
python -c \"import numpy; print(f'NumPy: {numpy.__version__}')\"\n# Expected: 2.x.x (Venv version), not 1.2x (System version)\n
- Verify GPU Access:
python -c \"import torch, jax; print(f'GPU: {torch.cuda.is_available()}'); print(f'JAX: {jax.devices()}')\"\n
"},{"location":"HPC/#managing-dependencies","title":"Managing Dependencies","text":"env/hpc/requirements.txt is auto-generated from pyproject.toml. To regenerate:
uv run scripts/hpc/export_requirements.py\n
Modules listed in env/hpc/modules.txt are automatically excluded from the pip requirements to save space and use HPC-optimized binaries.
"},{"location":"api/analysis/","title":"Analysis & Plotting Tools","text":"This guide outlines the tools available for analyzing experimental data and generating poster-quality visualizations for the Brittle Star project.
"},{"location":"api/analysis/#shared-configuration","title":"Shared Configuration","text":"All plotting scripts share a central configuration in scripts/plots/plot_config.py. This file defines: - Color Palette: A color-blind friendly, high-contrast palette for different architectures. - Typography: Consistent font sizes and styles tailored for A0 posters. - Markers: Shared visual indicators, such as the \u2605 used for best performers.
"},{"location":"api/analysis/#comparison-visualization","title":"Comparison Visualization","text":"The scripts/plots/analyze_comparisons.py script generates grouped bar charts comparing the performance of different architectures across various morphologies.
"},{"location":"api/analysis/#usage","title":"Usage","text":"Run the script from the root of the project, providing the path to your evaluation CSV:
# Basic usage (saves PNG and SVG to runs/evaluation/plots/)\nuv run python scripts/plots/analyze_comparisons.py path/to/results.csv\n\n# Advanced usage for Figma/Poster integration\nuv run python scripts/plots/analyze_comparisons.py path/to/results.csv \\\n --output_dir docs/assets/plots/ \\\n --font_size 30 \\\n --fig_width 14 \\\n --fig_height 10\n
"},{"location":"api/analysis/#cli-arguments","title":"CLI Arguments","text":" input_csv: (Required) Path to the CSV file containing evaluation results.--output_dir, -o: Directory where plots will be saved (default: runs/evaluation/plots).--show_titles: Include titles in the plots. Default is False, as titles are typically added natively in design tools like Figma.--font_size: Base font size in points (default: 28).--fig_width / --fig_height: Physical dimensions of the plot in inches. Match these to your Figma layout to maintain exact font sizes.
"},{"location":"api/analysis/#outputs","title":"Outputs","text":"The script generates four key plots, each saved as both .png and .svg: 1. Forward Velocity: Grouped bar chart (cm/s). 2. Accumulated Reward: Mean cumulative reward. 3. Success Rate: Target acquisition percentage. 4. Distance Remaining: Navigational accuracy.
"},{"location":"api/analysis/#convergence-analysis","title":"Convergence Analysis","text":"The scripts/plots/analyze_convergence.py script determines the convergence point of training runs.
"},{"location":"api/analysis/#usage_1","title":"Usage","text":"uv run python scripts/plots/analyze_convergence.py --output_dir runs/convergence/\n
"},{"location":"api/analysis/#configuration","title":"Configuration","text":" - File Mapping: The script uses hardcoded paths in the
FILE_MAPPING dictionary. Update these paths to point to your specific run evaluation files. - CLI Arguments: Supports the same
--show_titles, --font_size, and --fig_width/height flags as the comparison script.
"},{"location":"api/analysis/#outputs_1","title":"Outputs","text":"Generates three plots (PNG & SVG): 1. convergence_comparison: Grouped horizontal bar chart. 2. progress_reward_curves: Line plots of reward over time. 3. progress_velocity_curves: Line plots of velocity over time.
"},{"location":"api/analysis/#poster-integration-figma","title":"Poster Integration (Figma)","text":""},{"location":"api/analysis/#svg-scaling","title":"SVG & Scaling","text":"We recommend using the SVG outputs for poster design in Figma: 1. No Resolution Loss: SVGs are vector-based and will remain sharp at any size. 2. Native Text: Text in the SVG imports as native text layers in Figma. 3. Exact Font Matching: To ensure a 28pt font in the plot matches a 28pt font in your poster, set the --fig_width and --fig_height to match the physical dimensions of the plot box in your Figma layout. 4. Editable: You can \"Ungroup\" the SVG in Figma to manually move labels, adjust colors, or tweak individual bars.
"},{"location":"api/analysis/#image-placeholders","title":"Image Placeholders","text":"The comparison charts include light-gray square placeholders below the X-axis. These are designed as guides; in Figma, you can drop your morphology renders or illustrations directly on top of these squares.
"},{"location":"api/environment/","title":"Brittle star environment","text":""},{"location":"api/environment/#creation","title":"Creation","text":"The environment package contains a factory class BrittleStarEnvFactory that creates instances of the environment/morphologies/... It uses the configuration classes defined in env_config.py to create the instances.
"},{"location":"api/environment/#configuration","title":"Configuration","text":"The data classes in env_config have default values as stated in the tutorials. * MorphologyConfig: configuration for the morphology of the brittle star. Contains number of arms, number of segments per arm, and control mode. * ArenaConfig: configuration for the arena. Sets the size of the arena, whether to set the ground floor to sand, attach a target and sizes of the walls. * EnvConfig: configuration for the environment. These set shared settings such as camera locations, simulation time and the task.
"},{"location":"api/environment/#backend-and-task-enums","title":"Backend and Task enums","text":"The Backend enum specifies either an MJC or MJX backend. * MJC: runs on CPU * MJX: uses jax on the gpu
The Task enum specifies which task to use. 2 items are present: * DIRECTED_LOCOMOTION: move to a target location * LIGHT_ESCAPE: situation where the robot must move to a darker location
"},{"location":"api/evaluation/","title":"Checkpoint & Model Evaluation","text":"This guide covers how to evaluate trained brittle star models, with a focus on measuring defect tolerance (amputations) across different controller architectures.
"},{"location":"api/evaluation/#checkpoint-evaluation-during-training","title":"Checkpoint Evaluation (During Training)","text":"The PPOTrainer can automatically evaluate every saved checkpoint using the fast MJX backend. This is enabled via configuration.
"},{"location":"api/evaluation/#configuration","title":"Configuration","text":"In your experiment config or via CLI:
python scripts/train.py evaluation.evaluate_checkpoints=true evaluation.eval_max_steps=5000\n
Results are saved to runs/<run_dir>/metrics/checkpoint_evaluation.csv and synced to Weights & Biases if enabled.
"},{"location":"api/evaluation/#cross-model-fault-tolerance-analysis","title":"Cross-Model & Fault Tolerance Analysis","text":"To measure how well different controllers handle damage (amputations), use scripts/compare_models.py. This script performs a grid search over models x morphologies.
- Create or update a YAML file in
configs/evaluation. - Run the benchmark:
python scripts/compare_models.py evaluation=poster\n
The script will evaluate every combination of model and morphology for the specified number of episodes.
The results are saved to a CSV (default: metrics/model_comparison.csv).
"},{"location":"api/evaluation/#csv-schema","title":"CSV Schema","text":"Column Description model_path Path to the trained weights. architecture The morph_mode of the model (e.g., CENTRALIZED, RING). arm_0 ... arm_4 Number of segments in each arm slot (0 = amputated). num_active_arms Total number of arms with segments > 0. seed The episode seed. eval_return Accumulated shaped reward. approx_max_velocity Average velocity: (initial_dist - final_dist) / steps. reached_target Whether the robot finished within the success radius."},{"location":"api/evaluation/#post-hoc-checkpoint-scanning","title":"Post-hoc Checkpoint Scanning","text":"If you need to re-evaluate every saved checkpoint in a run (e.g., to generate a learning curve with different metrics):
python scripts/evaluate_checkpoints.py \\\n simulation.model_path=runs/<run_id>/final_model.flax \\\n evaluation.eval_max_steps=2000\n
This script scans the checkpoints/ directory of the specified run and evaluates every .flax file it finds using the model's training morphology.
"},{"location":"api/simulation/","title":"Simulation & Evaluation","text":"The simulation pipeline allows you to visualize trained models and evaluate their performance under various conditions.
"},{"location":"api/simulation/#overview","title":"Overview","text":"The simulation pipeline is metadata-driven. Training-specific configurations (morphology, arena, environment, etc.) are automatically loaded from the _metadata.yaml file associated with the model checkpoint.
"},{"location":"api/simulation/#basic-simulation","title":"Basic Simulation","text":"To simulate a model in the MuJoCo viewer:
uv run scripts/simulate.py simulation.model_path=runs/your_run/final_model.flax\n
"},{"location":"api/simulation/#amputation-morphology-overrides","title":"Amputation & Morphology Overrides","text":"You can test trained models on different morphologies (e.g., amputating legs) by providing a morphology override. The observations will be automatically padded up to the training morphology's dimensions:
uv run scripts/simulate.py \\\n simulation.model_path=runs/your_run/final_model.flax \\\n simulation.morphology_override=configs/morphology/3_arms.yaml\n
"},{"location":"api/simulation/#video-recording","title":"Video Recording","text":"Recording videos requires the [evaluation] extra:
uv run scripts/simulate.py \\\n simulation.model_path=runs/your_run/final_model.flax \\\n simulation.record_video=true \\\n simulation.max_steps=1000\n
Videos and evaluation metadata are stored in timestamped folders alongside the model: runs/your_run/final_model_evaluations/eval_<timestamp>/simulation.mp4
For batch evaluation and cross-model comparison, see the Evaluation Guide.
"},{"location":"api/tracking/","title":"Tracking & Monitoring","text":"This guide explains how to monitor your experiments using Weights & Biases (WandB) and TensorBoard.
"},{"location":"api/tracking/#weights-biases-wandb","title":"Weights & Biases (WandB)","text":"WandB is used for online synchronization and visualization of training metrics.
"},{"location":"api/tracking/#authorization","title":"Authorization","text":"Export your API key in your terminal to enable WandB synchronization:
export WANDB_API_KEY=your_copied_api_key_here\n
Alternatively, you can log in using the CLI:
uv run wandb login\n
"},{"location":"api/tracking/#enabling-tracking","title":"Enabling Tracking","text":"To enable online sync during a training run, set logging.track=true on the command line:
uv run python scripts/train.py logging.track=true\n
You can also configure your project and entity:
uv run python scripts/train.py \\\n logging.track=true \\\n logging.wandb_project_name=\"MyProject\" \\\n logging.wandb_entity=\"my-team\"\n
These can also be set in your configuration YAML file under the logging key.
"},{"location":"api/tracking/#local-monitoring-with-tensorboard","title":"Local Monitoring with TensorBoard","text":"All runs are recorded locally in the runs/ directory (or the directory specified in experiment.base_run_dir). You can view scalars and other metrics with TensorBoard:
tensorboard --logdir runs/\n
Access the interface at http://localhost:6006.
"},{"location":"api/tracking/#cli-exploration-tool","title":"CLI Exploration Tool","text":"For quick diagnostics or to export data to CSV without launching the full TensorBoard UI, you can use the explore_tensorboard.py script:
uv run python scripts/analysis/explore_tensorboard.py runs/your_run_name/\n
See the detailed description in /scripts/analysis/README.md.
"},{"location":"api/training/","title":"Training Models","text":"This guide covers how to configure and run training experiments for the Brittle Star project using Hydra-based configurations.
"},{"location":"api/training/#configuration","title":"Configuration","text":"The project uses a modular configuration system powered by Hydra. Instead of passing many command-line flags, you select and override configuration groups.
"},{"location":"api/training/#creating-a-custom-experiment","title":"Creating a Custom Experiment","text":" -
Create a new experiment file: Create a file at configs/experiment/my_experiment.yaml. You can copy an existing one as a template:
cp configs/experiment/base.yaml configs/experiment/my_experiment.yaml\n
-
Edit configs/experiment/my_experiment.yaml to set your experiment parameters:
# @package _global_\nexperiment:\n exp_name: \"my_custom_run\"\n seed: 42\n
"},{"location":"api/training/#training-execution","title":"Training Execution","text":"To start a training run with the default settings defined in configs/main_config.yaml:
uv run python scripts/train.py\n
"},{"location":"api/training/#using-a-custom-experiment-configuration","title":"Using a Custom Experiment Configuration","text":"To run with your custom experiment file:
uv run python scripts/train.py experiment=my_experiment\n
uv run python scripts/train.py ppo.learning_rate=0.001 ppo.num_envs=32 logging.track=true\n
"},{"location":"api/training/#evaluation-during-training","title":"Evaluation During Training","text":"By default, the trainer saves checkpoints but does not evaluate them. To enable automatic headless evaluation of every saved checkpoint, set evaluation.evaluate_checkpoints=true:
uv run python scripts/train.py evaluation.evaluate_checkpoints=true\n
For more details on evaluation metrics and comparison tools, see Evaluation.
For more details on tracking your experiments, see Tracking & Monitoring.
"},{"location":"design/actor-critic/","title":"Actor-Critic Architecture","text":"To process observations into actions, our controllers utilize an Actor-Critic architecture. Because we use Proximal Policy Optimization (PPO), the pipeline fundamentally requires separate networks for the policy (Actor) and the value estimation (Critic).
Centralized Architecture (Baseline)
This pipeline treats the agent as a single entity and uses standard Proximal Policy Optimization (PPO).
- Centralized Actor: Composed of two chained MLPs (Sensor $\\rightarrow$ Motor) passing a hidden state between them. The centralized sensor receives the concatenated global state vector of all limbs at once and processes it into a hidden state. The centralized motor receives this hidden state and outputs the joint offsets for all actuators simultaneously. This is mathematically equivalent to using one large MLP with hidden layers, but splitting makes the implementation easier by allowing us to reuse the same components for the decentralized modules.
- Centralized Critic: Composed of two sequential MLPs (Feature Extractor $\\rightarrow$ Critic). Because PPO evaluates the state-value function, this network only receives the concatenated global state vector (no actions). It outputs a single scalar estimating the expected future reward for the entire agent.
Our policy and value networks use separate input networks/feature extractors as advised by the SEL3 course assistants and the blog. For continuous actions this should allow better learning at a small cost.
graph TD\n Obs([Global Observation])\n\n Sens[Sensor]\n Act[Motor]\n OutAct([Action Distribution<br/>mean, log_std])\n\n Feat[Feature extractor]\n Crit[Critic]\n OutCrit([Value Estimate<br/>scalar])\n\n Obs --> Sens\n Obs --> Feat\n\n Sens -->|\"Hidden state\"| Act\n Feat -->|\"Hidden state\"| Crit\n\n Act --> OutAct\n Crit --> OutCrit
Decentralized Architecture
This pipeline utilizes the \"Centralized Training with Decentralized Execution\" principle, specifically the NerveNet-MLP variant.
- Decentralized Actor, split into three distinct models:
- Sensor: A local model at each node. It receives its local state plus the goal vector directly, processing them into an initial hidden state.
- Propagator: Nodes synchronously compute and exchange messages with connected neighbors for $N$ steps to update their hidden states. See communication.md for details.
- Motor: A local model uses its final updated hidden state to output the joint offset strictly for its own actuator.
- Centralized Critic: Composed of two sequential MLPs (Feature Extractor $\\rightarrow$ Critic). During training, it acts globally by taking the concatenated state vectors from all sensors to output a single, global state-value scalar evaluating the entire agent's pose.
To keep the implementation simple, we should use one critic per node in our architecture, but only a single, global critic for all nodes at once, for the following reasons:
- Credit Assignment Problem (Ha, 2017): The MuJoCo simulator provides an overall reward based on the brittle star movement progression, e.g. total distance travelled. Using an isolated critic for each node in the network would not allow to determine which local action contributed to the global success. A global critic solves this by evaluating the combined state of the agent at once.
- Implementation simplicity: Building a second decentralized message-passing graph for the critic (NerveNet-2) would require more coding. Using a standard MLP that concatenates all raw input vectors is much easier to program while mathematically equivalent.
graph TD\n Obs([Local Observation])\n\n Sens[Sensor]\n Prop[Propagator]\n Feat[Feature extractor]\n\n Mot[Motor]\n Crit[Critic]\n\n OutMot([Action Distribution<br/>mean, log_std])\n OutCrit([Value Estimate<br/>scalar])\n\n Obs --> Sens\n Sens -->|\"Hidden state\"| Prop\n Obs --> Feat\n\n Prop -->|\"Hidden state\"| Mot\n\n\n Feat -->|\"Hidden state\"| Crit\n\n Mot --> OutMot\n Crit --> OutCrit\n\n Prop -.->|\"message passing\"|Prop
"},{"location":"design/actor-critic/#implementation-details-network-depth","title":"Implementation Details (Network Depth)","text":"Inspired by: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/
The MLPs used in both pipelines are defined with specific hidden layer configurations to balance learning capability and computational cost. As of right now, though this might change as we make progress in our experiments, we use:
- Input Networks (Sensors & Feature Extractors): These networks map the raw state inputs to internal hidden states. They are configured as standard dense networks with 2 hidden layers of 64 nodes each (
[64, 64]) and utilize tanh activation functions. - Output Networks (Motors, Actors & Critics): The final output models are intentionally kept shallow. The Actor directly projects the hidden state to a continuous action distribution (
mean and log_std) using a single dense output layer (zero hidden layers) initialized orthogonally. The Critic functions similarly, mapping the hidden representation to a single scalar value.
Note: For the continuous action distributions outputted by the Motor, we explicitly use mean and log_std as advised by previous research to maintain learning stability.
References
- Ha, D. (2017, October 29). A Visual Guide to Evolution Strategies. \u5927\u30c8\u30ed \u30fb Machine Learning. https://blog.otoro.net/2017/10/29/visual-evolution-strategies/
- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. \u2018Proximal Policy Optimization Algorithms\u2019. arXiv:1707.06347. Preprint, arXiv, 28 August 2017. https://doi.org/10.48550/arXiv.1707.06347.
- Wang, Tingwu, Renjie Liao, Jimmy Ba, and S. Fidler. \u2018NerveNet: Learning Structured Policy with Graph Neural Networks\u2019. Conference paper presented at International Conference on Learning Representations. 15 February 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613.
"},{"location":"design/communication/","title":"Communication scheme (Message Passing)","text":"Remember our research question:
\"What is the impact of different levels of controller modularity on learning speed, coordination, and fault tolerance (e.g. amputations) in brittle-star-like robots trained with Reinforcement Learning?\"
To test decentralized modularity (such as arm-level or segment-level controllers), the various modules must be able to communicate with each other to achieve coordinated locomotion. This is accomplished through message passing in a Graph Neural Network (GNN)-like architecture. Two prominent communication styles from the literature are N-step NerveNet (Wang et al., 2018) and bottom-up top-down Shared Modular Policies (Huang et al., 2020).
We have chosen to apply one uniform communication style across all modular architectures, specifically opting for N-step NerveNet.
"},{"location":"design/communication/#rationale","title":"Rationale","text":"Initially, our idea was to equip arm-level controllers with NerveNet message passing and segment-level controllers with SMP. However, we evaluated that this introduces a threat to the validity of our research question. If we observe differences in performance, it would be impossible to determine whether the variance is caused by the level of modularity, or by the difference in the message passing scheme. To purely compare modularity, the communication scheme style must remain constant.
Second, we decided that NerveNet is a better fit for our research. The morphology of our brittle star contains cycles at the decentralized level (e.g., a ring of segments or arms around the body). NerveNet has proven to be robust for arbitrary structures, including graphs with cycles. SMP inherently expects a tree structure for its bottom-up and top-down pass. Applying SMP to a ring structure requires a workaround to break that cycle.
"},{"location":"design/communication/#limitations-and-alternatives","title":"Limitations and alternatives","text":"Choosing NerveNet introduces a scalability issue as the morphology grows. In NerveNet, a message advances only one segment or node per propagation step. When dealing with long arms (e.g., > 5 segments), this requires a large number of propagation steps to transmit information from one tip of an arm to another.
If we were to use SMP instead - which is possible - the inner states of nodes are shared across the entire graph in just two passes. For very large or long morphologies, this would be much more scalable.
By rejecting SMP, we accept that our model might learn slower or require more computational power for highly segmented, extended morphologies.
References
- Wang, Tingwu, Renjie Liao, Jimmy Ba, and S. Fidler. \u2018NerveNet: Learning Structured Policy with Graph Neural Networks\u2019. Conference paper presented at International Conference on Learning Representations. 15 February 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613.
- Huang, Wenlong, Igor Mordatch, and Deepak Pathak. \u2018One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control\u2019. arXiv:2007.04976. Preprint, arXiv, 9 July 2020. https://doi.org/10.48550/arXiv.2007.04976.
"},{"location":"design/controllers/","title":"Levels of modularity and topology","text":"The brittle star can be controlled at different levels. A monolithic controller processes all inputs and outputs at once, whereas modular controllers divide the brains across the body, inspired by the biology of brittle stars.
We define four architectures to compare:
- Centralized, monolithic: A single Multi Layer Perceptron per robot that receives all observations and outputs all actions.
- Fully connected arm-level: Each arm contains an MLP that processes the inputs for that arm, an MLP that processes the communicated inner-states, and an MLP that outputs the actions for that arm. One policy for these MLPs is shared across the arms. The controllers in each arm are connected to each other and form a fully connected graph. There is no central disk, but the controllers are fully connected.
- Ring arm-level: Identical setup to the fully connected arm-level, but the controllers are connected in a ring structure. This setup is considered less centralized than the fully connected graph.
- Segment-level: Each segment contains the three MLPs discussed above. The base segments, attached to the body, form a ring structure, with the remaining segments attached as extended \"strings\". Segments can only communicate with segments that are physically connected to it.
"},{"location":"design/controllers/#rationale","title":"Rationale","text":"To fairly compare decentralized modularity against centralized control, the decentralized models should not be allowed to contain a central organ acting as a bottleneck or coordinator. By removing the central disk in the decentralized models and replacing it with a ring topology, we closely approximate the biological reality of the brittle star and test a decentralized morphology.
The fully connected graph functions as an intermediate step in between a fully centralized and a decentralized ring. We use it to test whether our models scale to more complex structures.
"},{"location":"design/input_action_spaces/","title":"Input (state) and output (action) spaces","text":"To effectively learn locomotion and navigation, the agent requires a well-defined observation space (inputs) and action space (outputs). The control models map these observations directly to physical movements.
Inputs (state space)
The observation space provides the agent with its current physical state and its navigational objective. With a decentralized control architecture in mind, we divide these inputs into global and local states.
Global inputs, always broadcasted to all nodes:
- Vertical orientation/tilt: A single, simplified metric representing the tilt/vertical alignment of the agent's central body/disk, a.k.a. the deviation from the global Z-axis. Its value is derived from the environment's raw disk rotation 3D vector $[roll, pitch, yaw]$: $$ tilt = sqrt(roll^2 + pitch^2) $$
- Goal vector: A 2D unit vector representing the egocentric direction to the target. A value of $[1.0, 0.0]$ indicates that the target is directly in front of the agent (angle 0).
Local inputs, routed directly to specific nodes:
- Joint positions: The current angles of all joints within the morphology.
- Joint velocities: The current angular velocities of the joints.
- Joint actuator forces: The physical forces currently exerted at each specific joint.
- Segment contact: These values indicate whether each physical segment of the agent is currently touching the ground.
Outputs (action space)
The action space defines how the agent interacts with the environment.
- Joint offsets: absolute target positions (offsets) for the joints, i.e. the exact angle the joint should move to.
"},{"location":"design/input_action_spaces/#normalization-and-scaling","title":"Normalization and Scaling","text":"Both the input (observation) and output (action) spaces are rescaled to the range $[-1, 1]$.
For the input space, all raw physical values (angles, velocities, forces, distances) are normalized based on their defined physical bounds. If a value exceeds these bounds during simulation, it is clipped to the $[-1, 1]$ range.
For the output space, the neural network's tanh-activated outputs (which naturally fall in $[-1, 1]$) are linearly mapped to the physical joint limits defined in the robot's morphology.
"},{"location":"design/input_action_spaces/#rationale","title":"Rationale","text":"When designing the state space, we must ask: Could a human operator perform this task given only these inputs?
- Inclusion of Joint Velocities: Because our control models do not inherently possess memory of previous timesteps, providing only the joint position is insufficient to determine the direction a limb is currently moving. By explicitly including joint velocities, the agent can immediately infer momentum and movement direction without needing to memorize past states.
- Absolute Joint Offsets: The physical Brittle Star robot relies on servo motors (if we were to build this simulated robot), which are inherently position-controlled devices. (Continuous rotation servos exist, but they are less commonly used for joints.) If our network outputted continuous torques (forces), a significant portion of the reinforcement learning process would be wasted on learning low-level PID control dynamics (i.e., how much force to apply to hold a position). Abstracting this away forces the learning algorithm to focus entirely on higher-level gait generation and locomotion.
- Simplified vertical orientation: We drop the full 3D spatial rotation and angular velocity arrays in favor of a single vertical orientation metric (tilt). For a brittle star moving accross a flat plane, this metric is sufficient for the agent to sense if it is losing balance or flipping over.
- Force representation: We strictly retain the joint actuator forces and drop the more generic actuator force. Forces that are explicitly tied to individual joints are significantly easier to route into decentralized, local limb nodes, which is necessary for our message-passing architecture.
- Goal Vector (Distance + Angle): Providing only the scalar \"distance to the goal\" as an input is akin to blindfolding the robot and asking it to find a target by playing \"hot or cold.\" By providing a full vector, the agent knows exactly where the target is relative to its current orientation, allowing for directed and efficient locomotion.
The goal representation is explicitly divided into a directional vector and a scalar distance. Keeping the direction as a normalized unit vector bounds the values to the $[-1, 1]$ range, which stabilizes neural network training. Providing only a scalar \"distance to the goal\" would force the agent to learning localized searching behaviors (e.g. random walks or spiraling) to deduce the direction, drastically increasing the difficulty of the learning task.
NOTE: We later dropped the \"distance to vector\", switching to only a direction as the input. Our reasoning is the agent should always move towards the goal (it should not learn to stop at the goal), which allows for this simplification that decreases the model input size.
The environment provides a raw unit_xy_direction_to_target (global), which we transform into a calculated robot_direction_to_target (egocentric) before passing it to the MLPs. This vector consists of the X and Y direction, where a value of $[1.0, 0.0]$ (mapping to an angle of $0$) means the robot is facing directly towards the target. - Contact sensors: Segment contact detects external ground interaction and is biologically vital for timing gait transitions. - Zero-Centered Rescaling ($[-1, 1]$): Using a zero-centered range is standard best practice for continuous control tasks. It provides several mathematical and physical advantages: - Improved Gradient Flow: Neural networks optimize faster when inputs are zero-centered. If all inputs were positive (e.g., $[0, 1]$), the gradients during backpropagation would be forced to the same sign, causing inefficient \"zig-zag\" weight updates. - Meaningful Neutral State: In robotics, $0.0$ naturally represents a resting state (zero velocity, centered position, no force). In a $[-1, 1]$ system, this physical rest maps to a neutral $0.0$ signal in the network. This also correctly communicaties a \"neutral/dead\" signal for amputated limbs that are padded with $0.0$ values.
Specifically, we do not include some available inputs:
- Global position: Absolute spatial coordinates can cause the agent to overfit to a specific coordinate frame or map, rather than learning general, adaptable locomotion strategies.
"},{"location":"design/input_action_spaces/#limitations-and-alternatives","title":"Limitations and alternatives","text":"Alternative state and action formulations include:
- Torque-based continuous control: In many continuous control tasks (like standard MuJoCo benchmarks), actions represent continuous torques applied to joints. While this provides more granular, low-level physical control, it heavily complicates training and does not align well with the physical reality of servo-driven hardware.
- Recurrent Neural Networks (RNNs) / Frame Stacking: Instead of explicitly passing velocities in the state space, the network could infer momentum by observing a history of past states. Using RNNs or frame stacking allows the agent to build an internal memory of movement. However, this significantly increases architectural complexity and training time compared to explicitly providing the velocity data.
- Scalar Goal Distance: Giving the agent only the scalar distance to the target would force it to learn a localized searching behavior (e.g., spiraling or random walks) to determine the correct direction. While biologically plausible for simpler organisms following chemical gradients, it drastically increases the difficulty of the learning task.
- $[0, 1]$ Rescaling: While some domains (like computer vision) use $[0, 1]$ scaling, it is generally avoided in robotics. Scaling to $[0, 1]$ would mean that a resting joint (velocity = 0) maps to an input of $0.5$. This constant positive bias forces the network to waste capacity learning to ignore or subtract this baseline signal just to stand still. Furthermore, it breaks the \"dead signal\" interpretation of zero-padding used for amputations.
"},{"location":"design/input_action_spaces/#mujoco","title":"MuJoCo","text":"This is what the filtered input vectors look like in MuJoCo, with $J$ joints and $S$ segments:
joint_position: shape=(J,), dtype=float64joint_velocity: shape=(J,), dtype=float64joint_actuator_force: shape=(J,), dtype=float64segment_contact: shape=(S,), dtype=float64robot_direction_to_target: shape=(2,), dtype=float64, egocentricdisk_z_tilt: shape=(1,), dtype=float64, derived from disk_rotation
This brings the entire input space down to $3J + S + 4$ float64's, compared to $4J + S + 15$ float64's for the unfiltered inputs.
For reference, these are all the inputs that are available in the MuJoCo environment:
obs keys: ['joint_position', 'joint_velocity', 'joint_actuator_force', 'actuator_force', 'disk_position', 'disk_rotation', 'disk_linear_velocity', 'disk_angular_velocity', 'tendon_position', 'tendon_velocity', 'segment_contact', 'unit_xy_direction_to_target', 'xy_distance_to_target']\n\nraw observations dict:\n{'joint_position': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),\n 'joint_velocity': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),\n 'joint_actuator_force': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),\n 'actuator_force': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),\n 'disk_position': array([0. , 0. , 0.11]),\n 'disk_rotation': (0.0, -0.0, 0.0),\n 'disk_linear_velocity': array([0., 0., 0.]),\n 'disk_angular_velocity': array([0., 0., 0.]),\n 'tendon_position': array([], dtype=float64),\n 'tendon_velocity': array([], dtype=float64),\n 'segment_contact': array([0., 0., 0., 0., 0., 0.]),\n 'unit_xy_direction_to_target': array([-0.95333378, -0.30191837]),\n 'xy_distance_to_target': array([3.])}\n\n(shapes)\njoint_position: shape=(12,), dtype=float64, size=12\njoint_velocity: shape=(12,), dtype=float64, size=12\njoint_actuator_force: shape=(12,), dtype=float64, size=12\nactuator_force: shape=(12,), dtype=float64, size=12\ndisk_position: shape=(3,), dtype=float64, size=3\ndisk_rotation: shape=(3,), dtype=float64, size=3\ndisk_linear_velocity: shape=(3,), dtype=float64, size=3\ndisk_angular_velocity: shape=(3,), dtype=float64, size=3\ntendon_position: shape=(0,), dtype=float64, size=0\ntendon_velocity: shape=(0,), dtype=float64, size=0\nsegment_contact: shape=(6,), dtype=float64, size=6\nxy_distance_to_target: shape=(1,), dtype=float64, size=1\n
"},{"location":"design/learning_algorithm/","title":"Reinforcement Learning Algorithm","text":"To control the continuous action space (the joints of the robot) based on sensor data, we require a reliable Reinforcement Learning (RL) algorithm or optimization strategy.
We have chosen Proximal Policy Optimization (PPO) (Schulman et al., 2017).
"},{"location":"design/learning_algorithm/#rationale","title":"Rationale","text":"PPO is an on-policy algorithm known for its stability and robustness (safe training without excessive variance). More importantly, it requires relatively little hyperparameter tuning compared to other algorithms. Since NerveNet was successfully trained using PPO (Wang et al., 2018), selecting PPO significantly reduces the risk of convergence failures.
"},{"location":"design/learning_algorithm/#limitations-and-alternatives","title":"Limitations and alternatives","text":"Alternative learning algorithms include:
- Twin Delayed DDPG (Fujimoto et al., 2018): TD3 is a strong off-policy alternative used in the SMP paper (Huang et al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptations. However, this approach would be more complex and error-prone than with PPO.
- Evolution strategies (ES): Evolution strategies are useful for optimizing Central Pattern Generators (CPGs), e.g. CMA-ES, OpenAI-ES. While this method is easier to distribute and parallelize, ES typically scales worse with exceptionally large observation spaces compared to gradient-based RL methods like PPO.
References
- Fujimoto, Scott, Herke Hoof, and David Meger. \u2018Addressing Function Approximation Error in Actor-Critic Methods\u2019. Proceedings of the 35th International Conference on Machine Learning, 3 July 2018, 1587-96. https://proceedings.mlr.press/v80/fujimoto18a.html.
- Huang, Wenlong, Igor Mordatch, and Deepak Pathak. \u2018One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control\u2019. arXiv:2007.04976. Preprint, arXiv, 9 July 2020. https://doi.org/10.48550/arXiv.2007.04976.
- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. \u2018Proximal Policy Optimization Algorithms\u2019. arXiv:1707.06347. Preprint, arXiv, 28 August 2017. https://doi.org/10.48550/arXiv.1707.06347.
"},{"location":"design/reward_function/","title":"Reward function and observation space","text":"The robot needs to know whether its movements contribute to the ultimate goal of locomotion towards a target. Sensor inputs must be distributed fairly to guarantee an objective comparison between different architectures.
- The distance from the robot to the target and/or the light intensity are treated as global inputs.
- Positions and joints, which are normalized to floating-point values between 0 and 1, are considered local inputs.
- The reward function is centered around minimizing the distance to the goal or maximizing the movement towards the goal within a finite number of timesteps $T$.
- To motivate efficient movement, the amount of timesteps taken to reach the goal will be used as penalty.
- An extra penalty based on movement relative to the current step and the previous is used to penalize a movement away from the target.
"},{"location":"design/reward_function/#from-reward-to-ppo","title":"From reward to PPO","text":"The resulting reward is passed to our PPO library. Our critic network (value function) predicts how good our eventual reward will be for the current state, this value is combined with the reward from the reward function to get advantages. These advantages are then used to calculate the losses to update both our critic and actor pipeline.
"},{"location":"design/reward_function/#rationale","title":"Rationale","text":"Using a light source (or a gradient) is biologically plausible for many simple organisms. By normalizing all signals between 0 and 1, PPO training is highly stabilized. The timesteps must be finite to reset the environment in a timely manner if the policy gets stuck in a local minimum.
"},{"location":"design/reward_function/#limitations-and-alternatives","title":"Limitations and alternatives","text":"Providing global information to all individual decentralized segments can be considered biologically cheating or practically infeasible once the robot would be physically built. Some sensory input cannot be put in each joint, for example.
The alternative is to provide the global input to the outermost segments of the arms, or a specific set of segments assigned with this functionality. The network would then have to learn to propagate this signal throughout the body via message passing. While biologically more accurate, this drastically complicates the learning process. We have written this down as potential future research.
"},{"location":"scripts/analysis/","title":"Experiment Analysis Tools","text":"This directory contains scripts for post-processing and analyzing experiment results, including TensorBoard logs and saved model weights.
"},{"location":"scripts/analysis/#scripts","title":"Scripts","text":""},{"location":"scripts/analysis/#1-explore_tensorboardpy","title":"1. explore_tensorboard.py","text":"A CLI tool to summarize TensorBoard tfevents files without a GUI.
Key Features: - Displays last values, min, max, and step counts for all scalar metrics. - Calculates total run duration and estimated completion percentage. - Exports granular scalar data to CSV for analysis in Excel/Pandas.
Usage:
# General usage\npython explore_tensorboard.py <run_directory>\n\n# Exporting data\npython explore_tensorboard.py <run_directory> --csv data.csv\n
Requirements: - pandas - tensorboard - tensorflow-cpu (or tensorflow)
"},{"location":"src/experiment_logger/","title":"Experiment Logger","text":"A standardized, unified interface for logging experiments across multiple backends (WandB, TensorBoard, and Local Disk).
This library is designed to be a standalone package that decouples the logging logic from the core training routines in the brittle_star_project.
"},{"location":"src/experiment_logger/#quick-start","title":"Quick Start","text":"The recommended way to use the logger is through the get_logger() singleton:
from experiment_logger import UnifiedLogger, get_logger\n\n# Initialize at the start of your script (e.g., in train.py)\nlogger = UnifiedLogger(\n run_name=\"my_experiment_run\",\n config={\"learning_rate\": 3e-4},\n project_name=\"MyProject\",\n base_dir=\"runs\",\n use_wandb=True\n)\n\n# In other files, retrieve the initialized singleton:\n# logger = get_logger()\n\n# Log metrics (Scalar values, numpy scalars, or JAX types)\nlogger.log({\"loss\": 0.5, \"accuracy\": 0.98}, step=100)\n\n# Standard logging (Mirrored to disk and stdout)\nlogger.info(\"Training started\")\nlogger.warning(\"Learning rate is very high\")\n\n# Save checkpoints (Automatically synced to WandB as artifacts)\nlogger.save_checkpoint(params, step=5000)\n
"},{"location":"src/experiment_logger/#logger-classes","title":"Logger Classes","text":""},{"location":"src/experiment_logger/#unifiedlogger","title":"UnifiedLogger","text":"The full suite for production training. It manages: - WandB: Syncs metrics and uploads model checkpoints as artifacts. - TensorBoard: Writes events for local visualization. - Local Disk: Stores metrics in metrics.yaml and textual logs in run.log.
"},{"location":"src/experiment_logger/#simplelogger","title":"SimpleLogger","text":"A zero-dependency fallback that uses standard Python print() statements. Use this for standalone testing or minimal environments where you don't need persistent monitoring.
from experiment_logger import SimpleLogger\nlogger = SimpleLogger(run_name=\"test_run\")\n
"},{"location":"src/experiment_logger/#api-features","title":"API Features","text":""},{"location":"src/experiment_logger/#loggerprogress_bariterable-kwargs","title":"logger.progress_bar(iterable, **kwargs)","text":"A smart wrapper around tqdm that automatically detects its environment. - Interactive Terminal: Displays a normal progress bar. - Non-Interactive (HPC): Automatically disables the bar to prevent log file bloat in slurm.out.
"},{"location":"src/experiment_logger/#loggerlog_non_interactivemsg-str","title":"logger.log_non_interactive(msg: str)","text":"Prints a message only when running in non-interactive environments. Useful for high-level progress tracking (e.g., \"Epoch 5 Complete\") without interactive noise.
"},{"location":"src/experiment_logger/#loggersave_checkpointparams-step-prefixcheckpoint","title":"logger.save_checkpoint(params, step, prefix=\"checkpoint\")","text":"Saves model parameters using Flax serialization. - Local Location: runs/<run_name>/checkpoints/ - WandB Logic: Automatically uploads the .flax file as a model artifact for lineage tracking.
"}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"],"fields":{"title":{"boost":1000.0},"text":{"boost":1.0},"tags":{"boost":1000000.0}}},"docs":[{"location":"","title":"Documentation","text":""},{"location":"#design-architecture-design","title":"Design & architecture (/design)","text":"If you are interested in the \"why did you do it like this?\"
- Actor/critic architecture: Description of the actor-critic pipeline.
- Communication: Message propagation, Nerve-Net style.
- Controllers: Macroscopig brain toplogy, centralized, arm-level, segment-level.
- Input/output: Description of the model's input and output.
- Learning algorithm: RL techniques, i.e. PPO.
- Reward function: Goals, fitness tracking, and reward structures.
"},{"location":"#api-reference-api","title":"API reference (/api)","text":"If you are interested in the \"how do I use it?\"
- Training: How to configure and run experiments.
- Tracking & Monitoring: Setting up WandB and TensorBoard to monitor runs.
- Simulation: Visualizing and evaluating models.
- Environment: MuJoCo environment interaction and configuration.
- Analysis: Comparing checkpoints and generating plots.
- Evaluation: Evaluating checkpoints and comparing fault tolerance.
"},{"location":"CONTRIBUTING/","title":"Contribution Guidelines","text":"This document outlines the contribution protocols for the scientific software engineering project focusing on bio-inspired control architectures for brittle-star-like robots. The primary objective of this project is to produce scientific insight, rather than a commercial product.
"},{"location":"CONTRIBUTING/#1-scientific-context-methodology","title":"1. Scientific Context & Methodology","text":" - Research Focus: The goal is to study how controller modularity affects learning speed, coordination, and fault tolerance in brittle-star locomotion.
- Hypothesis-Driven Design: Clear hypotheses must dictate a structured methodology and rigorous evaluation. All design decisions must be formally documented prior to implementation.
- Scaffolding Approach: Development must start with simple setups before progressively increasing the complexity of environments and morphologies.
- Evaluation of Results: Negative results possess scientific validity when thoroughly analyzed. If a controller fails to learn locomotion, providing a comprehensive analysis of the failure is considered a strong scientific contribution.
- Reproducibility: Contributors must utilize fixed library versions. Configuration systems (such as json, gin, or yaml) must be employed to ensure reproducible runs.
"},{"location":"CONTRIBUTING/#2-clean-code-code-quality","title":"2. Clean Code & Code Quality","text":"Code readability is paramount, as code is read far more frequently than it is written.
- Naming Conventions: Variables and functions must utilize consistent, intention-revealing names. A long, descriptive name is strictly preferred over a short name accompanied by a comment.
- Function Design: Functions must be modular and adhere to the single responsibility principle. Arguments must be minimized, and boolean flag arguments controlling behavior should be avoided.
- Commenting: Code must document the \"how,\" while comments are strictly reserved for documenting the \"why\". Commented-out code is prohibited and must be deleted via version control.
- YAGNI: Contributors must adhere to the \"You Aren't Gonna Need It\" (YAGNI) principle and actively avoid premature optimization.
- Notebooks: Jupyter Notebooks are strictly limited to quick prototyping, tutorials, demonstrations, or post-processing analysis. They are explicitly forbidden for general software development because they discourage modularity.
"},{"location":"CONTRIBUTING/#3-version-control-repository-structure","title":"3. Version Control & Repository Structure","text":" - Git Practices: Commits must be frequent and small. Each commit should relate to exactly one piece of functionality.
- Branching Strategy: The
dev branch serves as the integration branch for pushing and merging code. Only stable releases may be pushed to the main branch. - Artifact Management: Data files, trained models, and large datasets must never be committed directly to Git. Git Large File Storage (LFS) must be used for tracking large files. All developers must have
git-lfs installed locally (see DEVELOPMENT.md for setup). - Repository Layout: The repository must maintain the following core directories:
src/ for algorithms, env/ for MuJoCo wrappers, config/ for experiment configurations, experiments/ for scripts, docs/ for Doxygen or ReadTheDocs documentation, and tests/ for unit tests.
"},{"location":"CONTRIBUTING/#4-architecture-tooling","title":"4. Architecture & Tooling","text":" - Algorithms & Frameworks: Proximal Policy Optimization (PPO) is the recommended baseline algorithm. CleanRL should be used as a starting point and adapted for continuous action spaces. All Artificial Neural Network (ANN) controller architectures must be implemented using Flax.
- Simulation: The simulation environment utilizes a MuJoCo brittle star. XML MuJoCo structures must remain realistic and respect morphological constraints.
- Experiment Tracking: Weights & Biases (wandb) must be utilized for tracking and logging all experiments.
- Code Styling: All code must conform to the chosen style guide (Google standard). This is enforced via
uv using ruff and pre-commit hooks.
"},{"location":"CONTRIBUTING/#5-ai-assisted-development-code-review","title":"5. AI-Assisted Development & Code Review","text":"This project supports AI-assisted development to enhance productivity, but contributors must take full responsibility for all AI-generated outputs.
- Self-Review Requirement: Contributors must thoroughly self-review all AI-assisted code, documentation, and configurations before requesting peer review. This includes verifying correctness, adherence to project standards, scientific validity, and integration with existing code.
- Quality Standards: AI-generated content must meet the same rigorous standards as manually written code, including proper testing, documentation, and alignment with the scientific methodology outlined in Section 1.
- Available Skills: This project provides specific AI skills for common tasks (located in
.agents/skills/), including linting and testing workflows. Contributors should leverage these skills to maintain consistency and quality. - Transparency: When using AI assistance for complex algorithmic decisions or scientific design choices, contributors should document the rationale in commit messages or code comments where appropriate.
"},{"location":"DEVELOPMENT/","title":"Development Guide","text":"This guide outlines how to set up the development environment for this project, prioritizing reproducible builds, environment parity, and cross-hardware compatibility.
"},{"location":"DEVELOPMENT/#reproducibility-uv","title":"Reproducibility & uv","text":"This project uses uv to manage dependencies and virtual environments. The uv.lock file is the absolute source of truth for package versions and must always be committed.
"},{"location":"DEVELOPMENT/#source-of-truth","title":"Source of Truth","text":" - Never modify
uv.lock manually. - To add a dependency, run
uv add <package>. - To update dependencies, run
uv lock --upgrade. - To sync your environment with the lockfile, run
uv sync --frozen.
"},{"location":"DEVELOPMENT/#git-lfs-critical","title":"Git LFS (Critical)","text":"All developers must have Git LFS installed locally. This repository tracks model weights (.pt, .safetensors, etc.), recordings (.mp4), and datasets using Git LFS.
- Setup: Run
git lfs install after cloning this repository. If you are using the .devcontainer or flake.nix, LFS is typically available automatically. - If you clone without LFS installed, run
git lfs pull after installation to fetch the actual data files instead of the small pointer files.
"},{"location":"DEVELOPMENT/#devcontainer-setup-recommended","title":"Devcontainer Setup (Recommended)","text":"The devcontainer provides an identical experience to local development but with all system dependencies pre-configured. It automatically detects your hardware (GPU vs CPU) and syncs the appropriate dependencies.
"},{"location":"DEVELOPMENT/#prerequisites","title":"Prerequisites","text":" - Docker Desktop or Docker Engine.
- NVIDIA Container Toolkit (for GPU support).
"},{"location":"DEVELOPMENT/#setup-for-vs-code","title":"Setup for VS Code","text":" - Install the Dev Containers extension.
- Open the project and click Reopen in Container.
- On first launch, the
post-create.sh script will: - Detect if an NVIDIA GPU is available via
nvidia-smi. - Run
uv sync --frozen --extra cuda if a GPU is found. - Run
uv sync --frozen otherwise. - The environment is stored in a named volume for
.venv to ensure persistence and performance.
"},{"location":"DEVELOPMENT/#setup-for-jetbrains-ides","title":"Setup for JetBrains IDEs","text":" - The IDE will detect the
.devcontainer/devcontainer.json file. - The environment is pre-configured to point to
/workspaces/project/.venv. - The hardware-aware sync will run automatically during container creation.
"},{"location":"DEVELOPMENT/#local-development-alternative","title":"Local Development (Alternative)","text":"If you prefer not to use Docker:
- Install uv.
- Run
uv sync --frozen (CPU) or uv sync --frozen --extra cuda (GPU).
"},{"location":"DEVELOPMENT/#hardware-acceleration-jax","title":"Hardware Acceleration (JAX)","text":"Verify your setup by running the JAX initialization test:
uv run pytest tests/test_jax_init.py\n
In the devcontainer, this will succeed on both CPU and GPU. A GpuDevice is expected if a GPU is detected and the cuda extra was installed.
"},{"location":"DEVELOPMENT/#logging-monitoring","title":"Logging & Monitoring","text":"This project uses a unified logging system through the experiment_logger package.
- Usage in Code: To use the logger in your scripts, refer to the package README for the API reference.
- WandB/TensorBoard Setup: For information on how to configure tracking for experiments, see the Tracking & Monitoring API Guide.
The logger automatically detects if it is running in an interactive terminal or a non-interactive environment (like an HPC Slurm job), adjusting progress bars and fallback modes accordingly.
"},{"location":"HPC/","title":"HPC Guide","text":"Full documentation: https://docs.hpc.ugent.be/
"},{"location":"HPC/#storage-overview","title":"Storage Overview","text":" - Run Outputs: Written to
$VSC_SCRATCH during the job (fast I/O) and copied to $VSC_DATA at the end for persistence. - Virtual Environments: Managed on
$VSC_DATA by mirroring configuration files. This avoids the 3GB home quota without requiring symlinks in the project root.
"},{"location":"HPC/#initial-environment-setup","title":"Initial Environment Setup","text":"Run once after cloning the repository. This script handles all modules, mirroring, and environment synchronization.
# Option A: Interactive (on a compute node)\nmodule swap cluster/donphan # Debug cluster (CPU only)\n# OR for GPU clusters:\n# module swap cluster/joltik\n# module swap cluster/accelgor\n# module swap cluster/litleo\n\nqsub -I -l nodes=1:gpus=1 # Only for GPU clusters\ncd \"${PBS_O_WORKDIR}\"\nbash scripts/hpc/install.sh\n\n# Option B: Batch (Run in background)\n# NOTE: GPU clusters (joltik/accelgor/litleo) require -l gpus=1 at runtime\nqsub -l gpus=1 scripts/hpc/install.sh\n
"},{"location":"HPC/#production-vs-debug-clusters","title":"Production vs. Debug Clusters","text":"Our scripts are cluster-agnostic and do not have hardcoded GPU requirements. Instead, you must request GPUs at runtime using the -l gpus=1 flag when submitting to a production GPU cluster.
"},{"location":"HPC/#debugging-donphan","title":"Debugging (Donphan)","text":"The donphan cluster does not support GPUs. Simply run the scripts without extra resource flags:
module swap cluster/donphan\nqsub scripts/hpc/train.pbs\n
"},{"location":"HPC/#production-joltik-accelgor-litleo","title":"Production (Joltik, Accelgor, Litleo)","text":"These clusters provide GPU acceleration and require a GPU request at runtime:
module swap cluster/joltik # or accelgor/litleo\nqsub -l gpus=1 scripts/hpc/train.pbs\n
"},{"location":"HPC/#interactive-debugging","title":"Interactive Debugging","text":"To activate your environment for interactive work, simply run the same install.sh script.
qsub -I -l nodes=1:ppn=4 -l walltime=1:00:00\ncd \"$PBS_O_WORKDIR\"\nbash scripts/hpc/install.sh\n
"},{"location":"HPC/#verification-commands","title":"Verification Commands","text":"After installation, run these commands to ensure your environment is set up correctly:
-
Verify Quota Safety:
ls -d venvs 2>/dev/null && echo \"FAIL\" || echo \">>> PASS: Project root is clean.\"\n
-
Verify Library Versions (NumPy Fix):
python -c \"import numpy; print(f'NumPy: {numpy.__version__}')\"\n# Expected: 2.x.x (Venv version), not 1.2x (System version)\n
-
Verify GPU Access:
python -c \"import torch, jax; print(f'GPU: {torch.cuda.is_available()}'); print(f'JAX: {jax.devices()}')\"\n
"},{"location":"HPC/#managing-dependencies","title":"Managing Dependencies","text":"env/hpc/requirements.txt is auto-generated from pyproject.toml. To regenerate:
uv run scripts/hpc/export_requirements.py\n
Modules listed in env/hpc/modules.txt are automatically excluded from the pip requirements to save space and use HPC-optimized binaries.
"},{"location":"api/analysis/","title":"Analysis & Plotting Tools","text":"This guide outlines the tools available for analyzing experimental data and generating poster-quality visualizations for the Brittle Star project.
"},{"location":"api/analysis/#shared-configuration","title":"Shared Configuration","text":"All plotting scripts share a central configuration in scripts/plots/plot_config.py. This file defines:
- Color Palette: A color-blind friendly, high-contrast palette for different architectures.
- Typography: Consistent font sizes and styles tailored for A0 posters.
- Markers: Shared visual indicators, such as the \u2605 used for best performers.
"},{"location":"api/analysis/#comparison-visualization","title":"Comparison Visualization","text":"The scripts/plots/analyze_comparisons.py script generates grouped bar charts comparing the performance of different architectures across various morphologies.
"},{"location":"api/analysis/#usage","title":"Usage","text":"Run the script from the root of the project, providing the path to your evaluation CSV:
# Basic usage (saves PNG and SVG to runs/evaluation/plots/)\nuv run python scripts/plots/analyze_comparisons.py path/to/results.csv\n\n# Advanced usage for Figma/Poster integration\nuv run python scripts/plots/analyze_comparisons.py path/to/results.csv \\\n --output_dir docs/assets/plots/ \\\n --font_size 30 \\\n --fig_width 14 \\\n --fig_height 10\n
"},{"location":"api/analysis/#cli-arguments","title":"CLI Arguments","text":" input_csv: (Required) Path to the CSV file containing evaluation results.--output_dir, -o: Directory where plots will be saved (default: runs/evaluation/plots).--show_titles: Include titles in the plots. Default is False, as titles are typically added natively in design tools like Figma.--font_size: Base font size in points (default: 28).--fig_width / --fig_height: Physical dimensions of the plot in inches. Match these to your Figma layout to maintain exact font sizes.
"},{"location":"api/analysis/#outputs","title":"Outputs","text":"The script generates four key plots, each saved as both .png and .svg:
- Forward Velocity: Grouped bar chart (cm/s).
- Accumulated Reward: Mean cumulative reward.
- Success Rate: Target acquisition percentage.
- Distance Remaining: Navigational accuracy.
"},{"location":"api/analysis/#convergence-analysis","title":"Convergence Analysis","text":"The scripts/plots/analyze_convergence.py script determines the convergence point of training runs.
"},{"location":"api/analysis/#usage_1","title":"Usage","text":"uv run python scripts/plots/analyze_convergence.py --output_dir runs/convergence/\n
"},{"location":"api/analysis/#configuration","title":"Configuration","text":" - File Mapping: The script uses hardcoded paths in the
FILE_MAPPING dictionary. Update these paths to point to your specific run evaluation files. - CLI Arguments: Supports the same
--show_titles, --font_size, and --fig_width/height flags as the comparison script.
"},{"location":"api/analysis/#outputs_1","title":"Outputs","text":"Generates three plots (PNG & SVG):
convergence_comparison: Grouped horizontal bar chart.progress_reward_curves: Line plots of reward over time.progress_velocity_curves: Line plots of velocity over time.
"},{"location":"api/analysis/#poster-integration-figma","title":"Poster Integration (Figma)","text":""},{"location":"api/analysis/#svg-scaling","title":"SVG & Scaling","text":"We recommend using the SVG outputs for poster design in Figma:
- No Resolution Loss: SVGs are vector-based and will remain sharp at any size.
- Native Text: Text in the SVG imports as native text layers in Figma.
- Exact Font Matching: To ensure a
28pt font in the plot matches a 28pt font in your poster, set the --fig_width and --fig_height to match the physical dimensions of the plot box in your Figma layout. - Editable: You can \"Ungroup\" the SVG in Figma to manually move labels, adjust colors, or tweak individual bars.
"},{"location":"api/analysis/#image-placeholders","title":"Image Placeholders","text":"The comparison charts include light-gray square placeholders below the X-axis. These are designed as guides; in Figma, you can drop your morphology renders or illustrations directly on top of these squares.
"},{"location":"api/environment/","title":"Brittle star environment","text":""},{"location":"api/environment/#creation","title":"Creation","text":"The environment package contains a factory class BrittleStarEnvFactory that creates instances of the environment/morphologies/... It uses the configuration classes defined in env_config.py to create the instances.
"},{"location":"api/environment/#configuration","title":"Configuration","text":"The data classes in env_config have default values as stated in the tutorials.
- MorphologyConfig: configuration for the morphology of the brittle star. Contains number of arms, number of segments per arm, and control mode.
- ArenaConfig: configuration for the arena. Sets the size of the arena, whether to set the ground floor to sand, attach a target and sizes of the walls.
- EnvConfig: configuration for the environment. These set shared settings such as camera locations, simulation time and the task.
"},{"location":"api/environment/#backend-and-task-enums","title":"Backend and Task enums","text":"The Backend enum specifies either an MJC or MJX backend.
- MJC: runs on CPU
- MJX: uses jax on the gpu
The Task enum specifies which task to use. 2 items are present:
- DIRECTED_LOCOMOTION: move to a target location
- LIGHT_ESCAPE: situation where the robot must move to a darker location
"},{"location":"api/evaluation/","title":"Checkpoint & Model Evaluation","text":"This guide covers how to evaluate trained brittle star models, with a focus on measuring defect tolerance (amputations) across different controller architectures.
"},{"location":"api/evaluation/#checkpoint-evaluation-during-training","title":"Checkpoint Evaluation (During Training)","text":"The PPOTrainer can automatically evaluate every saved checkpoint using the fast MJX backend. This is enabled via configuration.
"},{"location":"api/evaluation/#configuration","title":"Configuration","text":"In your experiment config or via CLI:
python scripts/train.py evaluation.evaluate_checkpoints=true evaluation.eval_max_steps=5000\n
Results are saved to runs/<run_dir>/metrics/checkpoint_evaluation.csv and synced to Weights & Biases if enabled.
"},{"location":"api/evaluation/#cross-model-fault-tolerance-analysis","title":"Cross-Model & Fault Tolerance Analysis","text":"To measure how well different controllers handle damage (amputations), use scripts/compare_models.py. This script performs a grid search over models x morphologies.
- Create or update a YAML file in
configs/evaluation. - Run the benchmark:
python scripts/compare_models.py evaluation=poster\n
The script will evaluate every combination of model and morphology for the specified number of episodes.
The results are saved to a CSV (default: metrics/model_comparison.csv).
"},{"location":"api/evaluation/#csv-schema","title":"CSV Schema","text":"Column Description model_path Path to the trained weights. architecture The morph_mode of the model (e.g., CENTRALIZED, RING). arm_0 ... arm_4 Number of segments in each arm slot (0 = amputated). num_active_arms Total number of arms with segments > 0. seed The episode seed. eval_return Accumulated shaped reward. approx_max_velocity Average velocity: (initial_dist - final_dist) / steps. reached_target Whether the robot finished within the success radius."},{"location":"api/evaluation/#post-hoc-checkpoint-scanning","title":"Post-hoc Checkpoint Scanning","text":"If you need to re-evaluate every saved checkpoint in a run (e.g., to generate a learning curve with different metrics):
python scripts/evaluate_checkpoints.py \\\n simulation.model_path=runs/<run_id>/final_model.flax \\\n evaluation.eval_max_steps=2000\n
This script scans the checkpoints/ directory of the specified run and evaluates every .flax file it finds using the model's training morphology.
"},{"location":"api/simulation/","title":"Simulation & Evaluation","text":"The simulation pipeline allows you to visualize trained models and evaluate their performance under various conditions.
"},{"location":"api/simulation/#overview","title":"Overview","text":"The simulation pipeline is metadata-driven. Training-specific configurations (morphology, arena, environment, etc.) are automatically loaded from the _metadata.yaml file associated with the model checkpoint.
"},{"location":"api/simulation/#basic-simulation","title":"Basic Simulation","text":"To simulate a model in the MuJoCo viewer:
uv run scripts/simulate.py simulation.model_path=runs/your_run/final_model.flax\n
"},{"location":"api/simulation/#amputation-morphology-overrides","title":"Amputation & Morphology Overrides","text":"You can test trained models on different morphologies (e.g., amputating legs) by providing a morphology override. The observations will be automatically padded up to the training morphology's dimensions:
uv run scripts/simulate.py \\\n simulation.model_path=runs/your_run/final_model.flax \\\n simulation.morphology_override=configs/morphology/3_arms.yaml\n
"},{"location":"api/simulation/#video-recording","title":"Video Recording","text":"Recording videos requires the [evaluation] extra:
uv run scripts/simulate.py \\\n simulation.model_path=runs/your_run/final_model.flax \\\n simulation.record_video=true \\\n simulation.max_steps=1000\n
Videos and evaluation metadata are stored in timestamped folders alongside the model: runs/your_run/final_model_evaluations/eval_<timestamp>/simulation.mp4
For batch evaluation and cross-model comparison, see the Evaluation Guide.
"},{"location":"api/tracking/","title":"Tracking & Monitoring","text":"This guide explains how to monitor your experiments using Weights & Biases (WandB) and TensorBoard.
"},{"location":"api/tracking/#weights-biases-wandb","title":"Weights & Biases (WandB)","text":"WandB is used for online synchronization and visualization of training metrics.
"},{"location":"api/tracking/#authorization","title":"Authorization","text":"Export your API key in your terminal to enable WandB synchronization:
export WANDB_API_KEY=your_copied_api_key_here\n
Alternatively, you can log in using the CLI:
uv run wandb login\n
"},{"location":"api/tracking/#enabling-tracking","title":"Enabling Tracking","text":"To enable online sync during a training run, set logging.track=true on the command line:
uv run python scripts/train.py logging.track=true\n
You can also configure your project and entity:
uv run python scripts/train.py \\\n logging.track=true \\\n logging.wandb_project_name=\"MyProject\" \\\n logging.wandb_entity=\"my-team\"\n
These can also be set in your configuration YAML file under the logging key.
"},{"location":"api/tracking/#local-monitoring-with-tensorboard","title":"Local Monitoring with TensorBoard","text":"All runs are recorded locally in the runs/ directory (or the directory specified in experiment.base_run_dir). You can view scalars and other metrics with TensorBoard:
tensorboard --logdir runs/\n
Access the interface at http://localhost:6006.
"},{"location":"api/tracking/#cli-exploration-tool","title":"CLI Exploration Tool","text":"For quick diagnostics or to export data to CSV without launching the full TensorBoard UI, you can use the explore_tensorboard.py script:
uv run python scripts/analysis/explore_tensorboard.py runs/your_run_name/\n
See the detailed description in /scripts/analysis/README.md.
"},{"location":"api/training/","title":"Training Models","text":"This guide covers how to configure and run training experiments for the Brittle Star project using Hydra-based configurations.
"},{"location":"api/training/#configuration","title":"Configuration","text":"The project uses a modular configuration system powered by Hydra. Instead of passing many command-line flags, you select and override configuration groups.
"},{"location":"api/training/#creating-a-custom-experiment","title":"Creating a Custom Experiment","text":" -
Create a new experiment file: Create a file at configs/experiment/my_experiment.yaml. You can copy an existing one as a template:
cp configs/experiment/base.yaml configs/experiment/my_experiment.yaml\n
-
Edit configs/experiment/my_experiment.yaml to set your experiment parameters:
# @package _global_\nexperiment:\n exp_name: \"my_custom_run\"\n seed: 42\n
"},{"location":"api/training/#training-execution","title":"Training Execution","text":"To start a training run with the default settings defined in configs/main_config.yaml:
uv run python scripts/train.py\n
"},{"location":"api/training/#using-a-custom-experiment-configuration","title":"Using a Custom Experiment Configuration","text":"To run with your custom experiment file:
uv run python scripts/train.py experiment=my_experiment\n
uv run python scripts/train.py ppo.learning_rate=0.001 ppo.num_envs=32 logging.track=true\n
"},{"location":"api/training/#evaluation-during-training","title":"Evaluation During Training","text":"By default, the trainer saves checkpoints but does not evaluate them. To enable automatic headless evaluation of every saved checkpoint, set evaluation.evaluate_checkpoints=true:
uv run python scripts/train.py evaluation.evaluate_checkpoints=true\n
For more details on evaluation metrics and comparison tools, see Evaluation.
For more details on tracking your experiments, see Tracking & Monitoring.
"},{"location":"design/actor-critic/","title":"Actor-Critic Architecture","text":"To process observations into actions, our controllers utilize an Actor-Critic architecture. Because we use Proximal Policy Optimization (PPO), the pipeline fundamentally requires separate networks for the policy (Actor) and the value estimation (Critic).
Centralized Architecture (Baseline)
This pipeline treats the agent as a single entity and uses standard Proximal Policy Optimization (PPO).
- Centralized Actor: Composed of two chained MLPs (Sensor $\\rightarrow$ Motor) passing a hidden state between them. The centralized sensor receives the concatenated global state vector of all limbs at once and processes it into a hidden state. The centralized motor receives this hidden state and outputs the joint offsets for all actuators simultaneously. This is mathematically equivalent to using one large MLP with hidden layers, but splitting makes the implementation easier by allowing us to reuse the same components for the decentralized modules.
- Centralized Critic: Composed of two sequential MLPs (Feature Extractor $\\rightarrow$ Critic). Because PPO evaluates the state-value function, this network only receives the concatenated global state vector (no actions). It outputs a single scalar estimating the expected future reward for the entire agent.
Our policy and value networks use separate input networks/feature extractors as advised by the SEL3 course assistants and the blog. For continuous actions this should allow better learning at a small cost.
graph TD\n Obs([Global Observation])\n\n Sens[Sensor]\n Act[Motor]\n OutAct([Action Distribution<br/>mean, log_std])\n\n Feat[Feature extractor]\n Crit[Critic]\n OutCrit([Value Estimate<br/>scalar])\n\n Obs --> Sens\n Obs --> Feat\n\n Sens -->|\"Hidden state\"| Act\n Feat -->|\"Hidden state\"| Crit\n\n Act --> OutAct\n Crit --> OutCrit
Decentralized Architecture
This pipeline utilizes the \"Centralized Training with Decentralized Execution\" principle, specifically the NerveNet-MLP variant.
- Decentralized Actor, split into three distinct models:
- Sensor: A local model at each node. It receives its local state plus the goal vector directly, processing them into an initial hidden state.
- Propagator: Nodes synchronously compute and exchange messages with connected neighbors for $N$ steps to update their hidden states. See communication.md for details.
- Motor: A local model uses its final updated hidden state to output the joint offset strictly for its own actuator.
- Centralized Critic: Composed of two sequential MLPs (Feature Extractor $\\rightarrow$ Critic). During training, it acts globally by taking the concatenated state vectors from all sensors to output a single, global state-value scalar evaluating the entire agent's pose.
To keep the implementation simple, we should use one critic per node in our architecture, but only a single, global critic for all nodes at once, for the following reasons:
- Credit Assignment Problem (Ha, 2017): The MuJoCo simulator provides an overall reward based on the brittle star movement progression, e.g. total distance travelled. Using an isolated critic for each node in the network would not allow to determine which local action contributed to the global success. A global critic solves this by evaluating the combined state of the agent at once.
- Implementation simplicity: Building a second decentralized message-passing graph for the critic (NerveNet-2) would require more coding. Using a standard MLP that concatenates all raw input vectors is much easier to program while mathematically equivalent.
graph TD\n Obs([Local Observation])\n\n Sens[Sensor]\n Prop[Propagator]\n Feat[Feature extractor]\n\n Mot[Motor]\n Crit[Critic]\n\n OutMot([Action Distribution<br/>mean, log_std])\n OutCrit([Value Estimate<br/>scalar])\n\n Obs --> Sens\n Sens -->|\"Hidden state\"| Prop\n Obs --> Feat\n\n Prop -->|\"Hidden state\"| Mot\n\n\n Feat -->|\"Hidden state\"| Crit\n\n Mot --> OutMot\n Crit --> OutCrit\n\n Prop -.->|\"message passing\"|Prop
"},{"location":"design/actor-critic/#implementation-details-network-depth","title":"Implementation Details (Network Depth)","text":"Inspired by: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/
The MLPs used in both pipelines are defined with specific hidden layer configurations to balance learning capability and computational cost. As of right now, though this might change as we make progress in our experiments, we use:
- Input Networks (Sensors & Feature Extractors): These networks map the raw state inputs to internal hidden states. They are configured as standard dense networks with 2 hidden layers of 64 nodes each (
[64, 64]) and utilize tanh activation functions. - Output Networks (Motors, Actors & Critics): The final output models are intentionally kept shallow. The Actor directly projects the hidden state to a continuous action distribution (
mean and log_std) using a single dense output layer (zero hidden layers) initialized orthogonally. The Critic functions similarly, mapping the hidden representation to a single scalar value.
Note: For the continuous action distributions outputted by the Motor, we explicitly use mean and log_std as advised by previous research to maintain learning stability.
References
- Ha, D. (2017, October 29). A Visual Guide to Evolution Strategies. \u5927\u30c8\u30ed \u30fb Machine Learning. https://blog.otoro.net/2017/10/29/visual-evolution-strategies/
- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. \u2018Proximal Policy Optimization Algorithms\u2019. arXiv:1707.06347. Preprint, arXiv, 28 August 2017. https://doi.org/10.48550/arXiv.1707.06347.
- Wang, Tingwu, Renjie Liao, Jimmy Ba, and S. Fidler. \u2018NerveNet: Learning Structured Policy with Graph Neural Networks\u2019. Conference paper presented at International Conference on Learning Representations. 15 February 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613.
"},{"location":"design/communication/","title":"Communication scheme (Message Passing)","text":"Remember our research question:
\"What is the impact of different levels of controller modularity on learning speed, coordination, and fault tolerance (e.g. amputations) in brittle-star-like robots trained with Reinforcement Learning?\"
To test decentralized modularity (such as arm-level or segment-level controllers), the various modules must be able to communicate with each other to achieve coordinated locomotion. This is accomplished through message passing in a Graph Neural Network (GNN)-like architecture. Two prominent communication styles from the literature are N-step NerveNet (Wang et al., 2018) and bottom-up top-down Shared Modular Policies (Huang et al., 2020).
We have chosen to apply one uniform communication style across all modular architectures, specifically opting for N-step NerveNet.
"},{"location":"design/communication/#rationale","title":"Rationale","text":"Initially, our idea was to equip arm-level controllers with NerveNet message passing and segment-level controllers with SMP. However, we evaluated that this introduces a threat to the validity of our research question. If we observe differences in performance, it would be impossible to determine whether the variance is caused by the level of modularity, or by the difference in the message passing scheme. To purely compare modularity, the communication scheme style must remain constant.
Second, we decided that NerveNet is a better fit for our research. The morphology of our brittle star contains cycles at the decentralized level (e.g., a ring of segments or arms around the body). NerveNet has proven to be robust for arbitrary structures, including graphs with cycles. SMP inherently expects a tree structure for its bottom-up and top-down pass. Applying SMP to a ring structure requires a workaround to break that cycle.
"},{"location":"design/communication/#limitations-and-alternatives","title":"Limitations and alternatives","text":"Choosing NerveNet introduces a scalability issue as the morphology grows. In NerveNet, a message advances only one segment or node per propagation step. When dealing with long arms (e.g., > 5 segments), this requires a large number of propagation steps to transmit information from one tip of an arm to another.
If we were to use SMP instead - which is possible - the inner states of nodes are shared across the entire graph in just two passes. For very large or long morphologies, this would be much more scalable.
By rejecting SMP, we accept that our model might learn slower or require more computational power for highly segmented, extended morphologies.
References
- Wang, Tingwu, Renjie Liao, Jimmy Ba, and S. Fidler. \u2018NerveNet: Learning Structured Policy with Graph Neural Networks\u2019. Conference paper presented at International Conference on Learning Representations. 15 February 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613.
- Huang, Wenlong, Igor Mordatch, and Deepak Pathak. \u2018One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control\u2019. arXiv:2007.04976. Preprint, arXiv, 9 July 2020. https://doi.org/10.48550/arXiv.2007.04976.
"},{"location":"design/controllers/","title":"Levels of modularity and topology","text":"The brittle star can be controlled at different levels. A monolithic controller processes all inputs and outputs at once, whereas modular controllers divide the brains across the body, inspired by the biology of brittle stars.
We define four architectures to compare:
- Centralized, monolithic: A single Multi Layer Perceptron per robot that receives all observations and outputs all actions.
- Fully connected arm-level: Each arm contains an MLP that processes the inputs for that arm, an MLP that processes the communicated inner-states, and an MLP that outputs the actions for that arm. One policy for these MLPs is shared across the arms. The controllers in each arm are connected to each other and form a fully connected graph. There is no central disk, but the controllers are fully connected.
- Ring arm-level: Identical setup to the fully connected arm-level, but the controllers are connected in a ring structure. This setup is considered less centralized than the fully connected graph.
- Segment-level: Each segment contains the three MLPs discussed above. The base segments, attached to the body, form a ring structure, with the remaining segments attached as extended \"strings\". Segments can only communicate with segments that are physically connected to it.
"},{"location":"design/controllers/#rationale","title":"Rationale","text":"To fairly compare decentralized modularity against centralized control, the decentralized models should not be allowed to contain a central organ acting as a bottleneck or coordinator. By removing the central disk in the decentralized models and replacing it with a ring topology, we closely approximate the biological reality of the brittle star and test a decentralized morphology.
The fully connected graph functions as an intermediate step in between a fully centralized and a decentralized ring. We use it to test whether our models scale to more complex structures.
"},{"location":"design/input_action_spaces/","title":"Input (state) and output (action) spaces","text":"To effectively learn locomotion and navigation, the agent requires a well-defined observation space (inputs) and action space (outputs). The control models map these observations directly to physical movements.
Inputs (state space)
The observation space provides the agent with its current physical state and its navigational objective. With a decentralized control architecture in mind, we divide these inputs into global and local states.
Global inputs, always broadcasted to all nodes:
- Vertical orientation/tilt: A single, simplified metric representing the tilt/vertical alignment of the agent's central body/disk, a.k.a. the deviation from the global Z-axis. Its value is derived from the environment's raw disk rotation 3D vector $[roll, pitch, yaw]$: $$ tilt = sqrt(roll^2 + pitch^2) $$
- Goal vector: A 2D unit vector representing the egocentric direction to the target. A value of $[1.0, 0.0]$ indicates that the target is directly in front of the agent (angle 0).
Local inputs, routed directly to specific nodes:
- Joint positions: The current angles of all joints within the morphology.
- Joint velocities: The current angular velocities of the joints.
- Joint actuator forces: The physical forces currently exerted at each specific joint.
- Segment contact: These values indicate whether each physical segment of the agent is currently touching the ground.
Outputs (action space)
The action space defines how the agent interacts with the environment.
- Joint offsets: absolute target positions (offsets) for the joints, i.e. the exact angle the joint should move to.
"},{"location":"design/input_action_spaces/#normalization-and-scaling","title":"Normalization and Scaling","text":"Both the input (observation) and output (action) spaces are rescaled to the range $[-1, 1]$.
For the input space, all raw physical values (angles, velocities, forces, distances) are normalized based on their defined physical bounds. If a value exceeds these bounds during simulation, it is clipped to the $[-1, 1]$ range.
For the output space, the neural network's tanh-activated outputs (which naturally fall in $[-1, 1]$) are linearly mapped to the physical joint limits defined in the robot's morphology.
"},{"location":"design/input_action_spaces/#rationale","title":"Rationale","text":"When designing the state space, we must ask: Could a human operator perform this task given only these inputs?
- Inclusion of Joint Velocities: Because our control models do not inherently possess memory of previous timesteps, providing only the joint position is insufficient to determine the direction a limb is currently moving. By explicitly including joint velocities, the agent can immediately infer momentum and movement direction without needing to memorize past states.
- Absolute Joint Offsets: The physical Brittle Star robot relies on servo motors (if we were to build this simulated robot), which are inherently position-controlled devices. (Continuous rotation servos exist, but they are less commonly used for joints.) If our network outputted continuous torques (forces), a significant portion of the reinforcement learning process would be wasted on learning low-level PID control dynamics (i.e., how much force to apply to hold a position). Abstracting this away forces the learning algorithm to focus entirely on higher-level gait generation and locomotion.
- Simplified vertical orientation: We drop the full 3D spatial rotation and angular velocity arrays in favor of a single vertical orientation metric (tilt). For a brittle star moving accross a flat plane, this metric is sufficient for the agent to sense if it is losing balance or flipping over.
- Force representation: We strictly retain the joint actuator forces and drop the more generic actuator force. Forces that are explicitly tied to individual joints are significantly easier to route into decentralized, local limb nodes, which is necessary for our message-passing architecture.
- Goal Vector (Distance + Angle): Providing only the scalar \"distance to the goal\" as an input is akin to blindfolding the robot and asking it to find a target by playing \"hot or cold.\" By providing a full vector, the agent knows exactly where the target is relative to its current orientation, allowing for directed and efficient locomotion.
The goal representation is explicitly divided into a directional vector and a scalar distance. Keeping the direction as a normalized unit vector bounds the values to the $[-1, 1]$ range, which stabilizes neural network training. Providing only a scalar \"distance to the goal\" would force the agent to learning localized searching behaviors (e.g. random walks or spiraling) to deduce the direction, drastically increasing the difficulty of the learning task.
NOTE: We later dropped the \"distance to vector\", switching to only a direction as the input. Our reasoning is the agent should always move towards the goal (it should not learn to stop at the goal), which allows for this simplification that decreases the model input size.
The environment provides a raw unit_xy_direction_to_target (global), which we transform into a calculated robot_direction_to_target (egocentric) before passing it to the MLPs. This vector consists of the X and Y direction, where a value of $[1.0, 0.0]$ (mapping to an angle of $0$) means the robot is facing directly towards the target. - Contact sensors: Segment contact detects external ground interaction and is biologically vital for timing gait transitions. - Zero-Centered Rescaling ($[-1, 1]$): Using a zero-centered range is standard best practice for continuous control tasks. It provides several mathematical and physical advantages: - Improved Gradient Flow: Neural networks optimize faster when inputs are zero-centered. If all inputs were positive (e.g., $[0, 1]$), the gradients during backpropagation would be forced to the same sign, causing inefficient \"zig-zag\" weight updates. - Meaningful Neutral State: In robotics, $0.0$ naturally represents a resting state (zero velocity, centered position, no force). In a $[-1, 1]$ system, this physical rest maps to a neutral $0.0$ signal in the network. This also correctly communicaties a \"neutral/dead\" signal for amputated limbs that are padded with $0.0$ values.
Specifically, we do not include some available inputs:
- Global position: Absolute spatial coordinates can cause the agent to overfit to a specific coordinate frame or map, rather than learning general, adaptable locomotion strategies.
"},{"location":"design/input_action_spaces/#limitations-and-alternatives","title":"Limitations and alternatives","text":"Alternative state and action formulations include:
- Torque-based continuous control: In many continuous control tasks (like standard MuJoCo benchmarks), actions represent continuous torques applied to joints. While this provides more granular, low-level physical control, it heavily complicates training and does not align well with the physical reality of servo-driven hardware.
- Recurrent Neural Networks (RNNs) / Frame Stacking: Instead of explicitly passing velocities in the state space, the network could infer momentum by observing a history of past states. Using RNNs or frame stacking allows the agent to build an internal memory of movement. However, this significantly increases architectural complexity and training time compared to explicitly providing the velocity data.
- Scalar Goal Distance: Giving the agent only the scalar distance to the target would force it to learn a localized searching behavior (e.g., spiraling or random walks) to determine the correct direction. While biologically plausible for simpler organisms following chemical gradients, it drastically increases the difficulty of the learning task.
- $[0, 1]$ Rescaling: While some domains (like computer vision) use $[0, 1]$ scaling, it is generally avoided in robotics. Scaling to $[0, 1]$ would mean that a resting joint (velocity = 0) maps to an input of $0.5$. This constant positive bias forces the network to waste capacity learning to ignore or subtract this baseline signal just to stand still. Furthermore, it breaks the \"dead signal\" interpretation of zero-padding used for amputations.
"},{"location":"design/input_action_spaces/#mujoco","title":"MuJoCo","text":"This is what the filtered input vectors look like in MuJoCo, with $J$ joints and $S$ segments:
joint_position: shape=(J,), dtype=float64joint_velocity: shape=(J,), dtype=float64joint_actuator_force: shape=(J,), dtype=float64segment_contact: shape=(S,), dtype=float64robot_direction_to_target: shape=(2,), dtype=float64, egocentricdisk_z_tilt: shape=(1,), dtype=float64, derived from disk_rotation
This brings the entire input space down to $3J + S + 4$ float64's, compared to $4J + S + 15$ float64's for the unfiltered inputs.
For reference, these are all the inputs that are available in the MuJoCo environment:
obs keys: ['joint_position', 'joint_velocity', 'joint_actuator_force', 'actuator_force', 'disk_position', 'disk_rotation', 'disk_linear_velocity', 'disk_angular_velocity', 'tendon_position', 'tendon_velocity', 'segment_contact', 'unit_xy_direction_to_target', 'xy_distance_to_target']\n\nraw observations dict:\n{'joint_position': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),\n 'joint_velocity': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),\n 'joint_actuator_force': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),\n 'actuator_force': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),\n 'disk_position': array([0. , 0. , 0.11]),\n 'disk_rotation': (0.0, -0.0, 0.0),\n 'disk_linear_velocity': array([0., 0., 0.]),\n 'disk_angular_velocity': array([0., 0., 0.]),\n 'tendon_position': array([], dtype=float64),\n 'tendon_velocity': array([], dtype=float64),\n 'segment_contact': array([0., 0., 0., 0., 0., 0.]),\n 'unit_xy_direction_to_target': array([-0.95333378, -0.30191837]),\n 'xy_distance_to_target': array([3.])}\n\n(shapes)\njoint_position: shape=(12,), dtype=float64, size=12\njoint_velocity: shape=(12,), dtype=float64, size=12\njoint_actuator_force: shape=(12,), dtype=float64, size=12\nactuator_force: shape=(12,), dtype=float64, size=12\ndisk_position: shape=(3,), dtype=float64, size=3\ndisk_rotation: shape=(3,), dtype=float64, size=3\ndisk_linear_velocity: shape=(3,), dtype=float64, size=3\ndisk_angular_velocity: shape=(3,), dtype=float64, size=3\ntendon_position: shape=(0,), dtype=float64, size=0\ntendon_velocity: shape=(0,), dtype=float64, size=0\nsegment_contact: shape=(6,), dtype=float64, size=6\nxy_distance_to_target: shape=(1,), dtype=float64, size=1\n
"},{"location":"design/learning_algorithm/","title":"Reinforcement Learning Algorithm","text":"To control the continuous action space (the joints of the robot) based on sensor data, we require a reliable Reinforcement Learning (RL) algorithm or optimization strategy.
We have chosen Proximal Policy Optimization (PPO) (Schulman et al., 2017).
"},{"location":"design/learning_algorithm/#rationale","title":"Rationale","text":"PPO is an on-policy algorithm known for its stability and robustness (safe training without excessive variance). More importantly, it requires relatively little hyperparameter tuning compared to other algorithms. Since NerveNet was successfully trained using PPO (Wang et al., 2018), selecting PPO significantly reduces the risk of convergence failures.
"},{"location":"design/learning_algorithm/#limitations-and-alternatives","title":"Limitations and alternatives","text":"Alternative learning algorithms include:
- Twin Delayed DDPG (Fujimoto et al., 2018): TD3 is a strong off-policy alternative used in the SMP paper (Huang et al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptations. However, this approach would be more complex and error-prone than with PPO.
- Evolution strategies (ES): Evolution strategies are useful for optimizing Central Pattern Generators (CPGs), e.g. CMA-ES, OpenAI-ES. While this method is easier to distribute and parallelize, ES typically scales worse with exceptionally large observation spaces compared to gradient-based RL methods like PPO.
References
- Fujimoto, Scott, Herke Hoof, and David Meger. \u2018Addressing Function Approximation Error in Actor-Critic Methods\u2019. Proceedings of the 35th International Conference on Machine Learning, 3 July 2018, 1587-96. https://proceedings.mlr.press/v80/fujimoto18a.html.
- Huang, Wenlong, Igor Mordatch, and Deepak Pathak. \u2018One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control\u2019. arXiv:2007.04976. Preprint, arXiv, 9 July 2020. https://doi.org/10.48550/arXiv.2007.04976.
- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. \u2018Proximal Policy Optimization Algorithms\u2019. arXiv:1707.06347. Preprint, arXiv, 28 August 2017. https://doi.org/10.48550/arXiv.1707.06347.
"},{"location":"design/reward_function/","title":"Reward function and observation space","text":"The robot needs to know whether its movements contribute to the ultimate goal of locomotion towards a target. Sensor inputs must be distributed fairly to guarantee an objective comparison between different architectures.
- The distance from the robot to the target and/or the light intensity are treated as global inputs.
- Positions and joints, which are normalized to floating-point values between 0 and 1, are considered local inputs.
- The reward function is centered around minimizing the distance to the goal or maximizing the movement towards the goal within a finite number of timesteps $T$.
- To motivate efficient movement, the amount of timesteps taken to reach the goal will be used as penalty.
- An extra penalty based on movement relative to the current step and the previous is used to penalize a movement away from the target.
"},{"location":"design/reward_function/#from-reward-to-ppo","title":"From reward to PPO","text":"The resulting reward is passed to our PPO library. Our critic network (value function) predicts how good our eventual reward will be for the current state, this value is combined with the reward from the reward function to get advantages. These advantages are then used to calculate the losses to update both our critic and actor pipeline.
"},{"location":"design/reward_function/#rationale","title":"Rationale","text":"Using a light source (or a gradient) is biologically plausible for many simple organisms. By normalizing all signals between 0 and 1, PPO training is highly stabilized. The timesteps must be finite to reset the environment in a timely manner if the policy gets stuck in a local minimum.
"},{"location":"design/reward_function/#limitations-and-alternatives","title":"Limitations and alternatives","text":"Providing global information to all individual decentralized segments can be considered biologically cheating or practically infeasible once the robot would be physically built. Some sensory input cannot be put in each joint, for example.
The alternative is to provide the global input to the outermost segments of the arms, or a specific set of segments assigned with this functionality. The network would then have to learn to propagate this signal throughout the body via message passing. While biologically more accurate, this drastically complicates the learning process. We have written this down as potential future research.
"},{"location":"scripts/analysis/","title":"Experiment Analysis Tools","text":"This directory contains scripts for post-processing and analyzing experiment results, including TensorBoard logs and saved model weights.
"},{"location":"scripts/analysis/#scripts","title":"Scripts","text":""},{"location":"scripts/analysis/#1-explore_tensorboardpy","title":"1. explore_tensorboard.py","text":"A CLI tool to summarize TensorBoard tfevents files without a GUI.
Key Features: - Displays last values, min, max, and step counts for all scalar metrics. - Calculates total run duration and estimated completion percentage. - Exports granular scalar data to CSV for analysis in Excel/Pandas.
Usage:
# General usage\npython explore_tensorboard.py <run_directory>\n\n# Exporting data\npython explore_tensorboard.py <run_directory> --csv data.csv\n
Requirements: - pandas - tensorboard - tensorflow-cpu (or tensorflow)
"},{"location":"src/experiment_logger/","title":"Experiment Logger","text":"A standardized, unified interface for logging experiments across multiple backends (WandB, TensorBoard, and Local Disk).
This library is designed to be a standalone package that decouples the logging logic from the core training routines in the brittle_star_project.
"},{"location":"src/experiment_logger/#quick-start","title":"Quick Start","text":"The recommended way to use the logger is through the get_logger() singleton:
from experiment_logger import UnifiedLogger, get_logger\n\n# Initialize at the start of your script (e.g., in train.py)\nlogger = UnifiedLogger(\n run_name=\"my_experiment_run\",\n config={\"learning_rate\": 3e-4},\n project_name=\"MyProject\",\n base_dir=\"runs\",\n use_wandb=True\n)\n\n# In other files, retrieve the initialized singleton:\n# logger = get_logger()\n\n# Log metrics (Scalar values, numpy scalars, or JAX types)\nlogger.log({\"loss\": 0.5, \"accuracy\": 0.98}, step=100)\n\n# Standard logging (Mirrored to disk and stdout)\nlogger.info(\"Training started\")\nlogger.warning(\"Learning rate is very high\")\n\n# Save checkpoints (Automatically synced to WandB as artifacts)\nlogger.save_checkpoint(params, step=5000)\n
"},{"location":"src/experiment_logger/#logger-classes","title":"Logger Classes","text":""},{"location":"src/experiment_logger/#unifiedlogger","title":"UnifiedLogger","text":"The full suite for production training. It manages: - WandB: Syncs metrics and uploads model checkpoints as artifacts. - TensorBoard: Writes events for local visualization. - Local Disk: Stores metrics in metrics.yaml and textual logs in run.log.
"},{"location":"src/experiment_logger/#simplelogger","title":"SimpleLogger","text":"A zero-dependency fallback that uses standard Python print() statements. Use this for standalone testing or minimal environments where you don't need persistent monitoring.
from experiment_logger import SimpleLogger\nlogger = SimpleLogger(run_name=\"test_run\")\n
"},{"location":"src/experiment_logger/#api-features","title":"API Features","text":""},{"location":"src/experiment_logger/#loggerprogress_bariterable-kwargs","title":"logger.progress_bar(iterable, **kwargs)","text":"A smart wrapper around tqdm that automatically detects its environment. - Interactive Terminal: Displays a normal progress bar. - Non-Interactive (HPC): Automatically disables the bar to prevent log file bloat in slurm.out.
"},{"location":"src/experiment_logger/#loggerlog_non_interactivemsg-str","title":"logger.log_non_interactive(msg: str)","text":"Prints a message only when running in non-interactive environments. Useful for high-level progress tracking (e.g., \"Epoch 5 Complete\") without interactive noise.
"},{"location":"src/experiment_logger/#loggersave_checkpointparams-step-prefixcheckpoint","title":"logger.save_checkpoint(params, step, prefix=\"checkpoint\")","text":"Saves model parameters using Flax serialization. - Local Location: runs/<run_name>/checkpoints/ - WandB Logic: Automatically uploads the .flax file as a model artifact for lineage tracking.
"}]}
\ No newline at end of file