Merge branch 'dev' into docfix
This commit is contained in:
JibrilExe
commit
f5a823c31e
80 changed files with 5260 additions and 1193 deletions
|
|
@ -62,22 +62,9 @@ In the devcontainer, this will succeed on both CPU and GPU. A `GpuDevice` is exp
|
|||
|
||||
## Logging & Monitoring
|
||||
|
||||
This project uses a unified logging system through the `experiment_logger` package. For a full API reference, see the [package README](../src/experiment_logger/README.md).
|
||||
|
||||
### Quick Setup
|
||||
|
||||
1. **Authorization**: Export your API key in your terminal to enable WandB synchronization:
|
||||
```bash
|
||||
export WANDB_API_KEY=your_copied_api_key_here
|
||||
```
|
||||
2. **Toggle Tracking**: Use the `--track` flag in `scripts/train.py` to enable online sync.
|
||||
3. **Local Monitoring**: All runs are recorded in the `runs/` directory. View scalars with TensorBoard:
|
||||
```bash
|
||||
tensorboard --logdir runs/
|
||||
```
|
||||
|
||||
### Environment Awareness
|
||||
|
||||
The logger automatically detects if it is running in an interactive terminal or a non-interactive environment (like an HPC Slurm job). It will automatically disable progress bars and switch to robust fallback modes (offline logging) to ensure your experiments never hang.
|
||||
This project uses a unified logging system through the `experiment_logger` package.
|
||||
|
||||
- **Usage in Code**: To use the logger in your scripts, refer to the [package README](../src/experiment_logger/README.md) for the API reference.
|
||||
- **WandB/TensorBoard Setup**: For information on how to configure tracking for experiments, see the [Tracking & Monitoring API Guide](./api/tracking.md).
|
||||
|
||||
The logger automatically detects if it is running in an interactive terminal or a non-interactive environment (like an HPC Slurm job), adjusting progress bars and fallback modes accordingly.
|
||||
|
|
|
|||
|
|
@ -2,6 +2,8 @@
|
|||
|
||||
## Design & architecture (`/design`)
|
||||
|
||||
If you are interested in the "why did you do it like this?"
|
||||
|
||||
- [Actor/critic architecture](./design/actor-critic.md): Description of the actor-critic pipeline.
|
||||
- [Communication](./design/communication.md): Message propagation, Nerve-Net style.
|
||||
- [Controllers](./design/controllers.md): Macroscopig brain toplogy, centralized, arm-level, segment-level.
|
||||
|
|
@ -11,5 +13,9 @@
|
|||
|
||||
## API reference (`/api`)
|
||||
|
||||
- [Environment](./api/environment.md): MuJoCo environment interaction, state retrieval, and configuration.
|
||||
- [Simulate](./api/simulate.md): Simulation rendering.
|
||||
If you are interested in the "how do I use it?"
|
||||
|
||||
- [Training](./api/training.md): How to configure and run experiments.
|
||||
- [Tracking & Monitoring](./api/tracking.md): Setting up WandB and TensorBoard to monitor runs.
|
||||
- [Simulation](./api/simulation.md): Visualizing and evaluating models.
|
||||
- [Environment](./api/environment.md): MuJoCo environment interaction and configuration.
|
||||
|
|
|
|||
84
docs/api/analysis.md
Normal file
84
docs/api/analysis.md
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
# Analysis & Plotting Tools
|
||||
|
||||
This guide outlines the tools available for analyzing experimental data and generating poster-quality visualizations for the Brittle Star project.
|
||||
|
||||
## Shared Configuration
|
||||
|
||||
All plotting scripts share a central configuration in `scripts/plots/plot_config.py`. This file defines:
|
||||
- **Color Palette:** A color-blind friendly, high-contrast palette for different architectures.
|
||||
- **Typography:** Consistent font sizes and styles tailored for A0 posters.
|
||||
- **Markers:** Shared visual indicators, such as the ★ used for best performers.
|
||||
|
||||
## Comparison Visualization
|
||||
|
||||
The `scripts/plots/analyze_comparisons.py` script generates grouped bar charts comparing the performance of different architectures across various morphologies.
|
||||
|
||||
### Usage
|
||||
|
||||
Run the script from the root of the project, providing the path to your evaluation CSV:
|
||||
|
||||
```bash
|
||||
# Basic usage (saves PNG and SVG to runs/evaluation/plots/)
|
||||
uv run python scripts/plots/analyze_comparisons.py path/to/results.csv
|
||||
|
||||
# Advanced usage for Figma/Poster integration
|
||||
uv run python scripts/plots/analyze_comparisons.py path/to/results.csv \
|
||||
--output_dir docs/assets/plots/ \
|
||||
--font_size 30 \
|
||||
--fig_width 14 \
|
||||
--fig_height 10
|
||||
```
|
||||
|
||||
### CLI Arguments
|
||||
|
||||
- `input_csv`: (Required) Path to the CSV file containing evaluation results.
|
||||
- `--output_dir`, `-o`: Directory where plots will be saved (default: `runs/evaluation/plots`).
|
||||
- `--show_titles`: Include titles in the plots. Default is **False**, as titles are typically added natively in design tools like Figma.
|
||||
- `--font_size`: Base font size in points (default: 28).
|
||||
- `--fig_width` / `--fig_height`: Physical dimensions of the plot in inches. Match these to your Figma layout to maintain exact font sizes.
|
||||
|
||||
### Outputs
|
||||
|
||||
The script generates four key plots, each saved as both `.png` and `.svg`:
|
||||
1. **Forward Velocity:** Grouped bar chart (cm/s).
|
||||
2. **Accumulated Reward:** Mean cumulative reward.
|
||||
3. **Success Rate:** Target acquisition percentage.
|
||||
4. **Distance Remaining:** Navigational accuracy.
|
||||
|
||||
---
|
||||
|
||||
## Convergence Analysis
|
||||
|
||||
The `scripts/plots/analyze_convergence.py` script determines the convergence point of training runs.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
uv run python scripts/plots/analyze_convergence.py --output_dir runs/convergence/
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
- **File Mapping:** The script uses hardcoded paths in the `FILE_MAPPING` dictionary. Update these paths to point to your specific run evaluation files.
|
||||
- **CLI Arguments:** Supports the same `--show_titles`, `--font_size`, and `--fig_width/height` flags as the comparison script.
|
||||
|
||||
### Outputs
|
||||
|
||||
Generates three plots (PNG & SVG):
|
||||
1. `convergence_comparison`: Grouped horizontal bar chart.
|
||||
2. `progress_reward_curves`: Line plots of reward over time.
|
||||
3. `progress_velocity_curves`: Line plots of velocity over time.
|
||||
|
||||
---
|
||||
|
||||
## Poster Integration (Figma)
|
||||
|
||||
### SVG & Scaling
|
||||
We recommend using the **SVG** outputs for poster design in Figma:
|
||||
1. **No Resolution Loss:** SVGs are vector-based and will remain sharp at any size.
|
||||
2. **Native Text:** Text in the SVG imports as native text layers in Figma.
|
||||
3. **Exact Font Matching:** To ensure a `28pt` font in the plot matches a `28pt` font in your poster, set the `--fig_width` and `--fig_height` to match the physical dimensions of the plot box in your Figma layout.
|
||||
4. **Editable:** You can "Ungroup" the SVG in Figma to manually move labels, adjust colors, or tweak individual bars.
|
||||
|
||||
### Image Placeholders
|
||||
The comparison charts include light-gray square placeholders below the X-axis. These are designed as guides; in Figma, you can drop your morphology renders or illustrations directly on top of these squares.
|
||||
56
docs/api/evaluation.md
Normal file
56
docs/api/evaluation.md
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
# Checkpoint & Model Evaluation
|
||||
|
||||
This guide covers how to evaluate trained brittle star models, with a focus on measuring defect tolerance (amputations) across different controller architectures.
|
||||
|
||||
## Checkpoint Evaluation (During Training)
|
||||
|
||||
The `PPOTrainer` can automatically evaluate every saved checkpoint using the fast MJX backend. This is enabled via configuration.
|
||||
|
||||
### Configuration
|
||||
|
||||
In your experiment config or via CLI:
|
||||
```bash
|
||||
python scripts/train.py evaluation.evaluate_checkpoints=true evaluation.eval_max_steps=5000
|
||||
```
|
||||
|
||||
Results are saved to `runs/<run_dir>/metrics/checkpoint_evaluation.csv` and synced to Weights & Biases if enabled.
|
||||
|
||||
## Cross-Model & Defect Tolerance Analysis
|
||||
|
||||
To measure how well different controllers handle damage (amputations), use `scripts/compare_models.py`. This script performs a grid search over models x morphologies.
|
||||
|
||||
1. Create or update a YAML file in `configs/evaluation`.
|
||||
2. Run the benchmark:
|
||||
|
||||
```bash
|
||||
python scripts/compare_models.py evaluation=poster
|
||||
```
|
||||
|
||||
The script will evaluate every combination of model and morphology for the specified number of episodes.
|
||||
|
||||
The results are saved to a CSV (default: `metrics/model_comparison.csv`).
|
||||
|
||||
### CSV Schema
|
||||
|
||||
| Column | Description |
|
||||
|-----------------------|--------------------------------------------------------------|
|
||||
| `model_path` | Path to the trained weights. |
|
||||
| `architecture` | The `morph_mode` of the model (e.g., `CENTRALIZED`, `RING`). |
|
||||
| `arm_0` ... `arm_4` | Number of segments in each arm slot (0 = amputated). |
|
||||
| `num_active_arms` | Total number of arms with segments > 0. |
|
||||
| `seed` | The episode seed. |
|
||||
| `eval_return` | Accumulated shaped reward. |
|
||||
| `approx_max_velocity` | Average velocity: `(initial_dist - final_dist) / steps`. |
|
||||
| `reached_target` | Whether the robot finished within the success radius. |
|
||||
|
||||
## Post-hoc Checkpoint Scanning
|
||||
|
||||
If you need to re-evaluate every saved checkpoint in a run (e.g., to generate a learning curve with different metrics):
|
||||
|
||||
```bash
|
||||
python scripts/evaluate_checkpoints.py \
|
||||
simulation.model_path=runs/<run_id>/final_model.flax \
|
||||
evaluation.eval_max_steps=2000
|
||||
```
|
||||
|
||||
This script scans the `checkpoints/` directory of the specified run and evaluates every `.flax` file it finds using the model's training morphology.
|
||||
|
|
@ -1,14 +0,0 @@
|
|||
# Training and Simulation for Brittle Star Models
|
||||
|
||||
## Simulating a model
|
||||
|
||||
In order to simulate and view the behavior of a trained model, you can use the `simulate.py` script. This script allows you to specify the path to a trained model and will launch a simulation using that model. This script has the following parameters:
|
||||
|
||||
- `--model`: The path to the trained model artifact to simulate.
|
||||
- `--model-type`: The type of model to simulate (e.g., `random`, ...)
|
||||
- `--task`: The task to simulate (e.g., `directed_locomotion`, ...)
|
||||
- `--seed`: The random seed for reproducibility.
|
||||
|
||||
```bash
|
||||
python simulate.py --model artifacts/my_model --model-type random --task directed_locomotion --seed 0
|
||||
```
|
||||
41
docs/api/simulation.md
Normal file
41
docs/api/simulation.md
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
# Simulation & Evaluation
|
||||
|
||||
The simulation pipeline allows you to visualize trained models and evaluate their performance under various conditions.
|
||||
|
||||
## Overview
|
||||
|
||||
The simulation pipeline is metadata-driven. Training-specific configurations (morphology, arena, environment, etc.) are automatically loaded from the `_metadata.yaml` file associated with the model checkpoint.
|
||||
|
||||
## Basic Simulation
|
||||
|
||||
To simulate a model in the MuJoCo viewer:
|
||||
|
||||
```bash
|
||||
uv run scripts/simulate.py simulation.model_path=runs/your_run/final_model.flax
|
||||
```
|
||||
|
||||
## Amputation & Morphology Overrides
|
||||
|
||||
You can test trained models on different morphologies (e.g., amputating legs) by providing a morphology override. The observations will be automatically padded up to the training morphology's dimensions:
|
||||
|
||||
```bash
|
||||
uv run scripts/simulate.py \
|
||||
simulation.model_path=runs/your_run/final_model.flax \
|
||||
simulation.morphology_override=configs/morphology/3_arms.yaml
|
||||
```
|
||||
|
||||
## Video Recording
|
||||
|
||||
Recording videos requires the `[evaluation]` extra:
|
||||
|
||||
```bash
|
||||
uv run scripts/simulate.py \
|
||||
simulation.model_path=runs/your_run/final_model.flax \
|
||||
simulation.record_video=true \
|
||||
simulation.max_steps=1000
|
||||
```
|
||||
|
||||
Videos and evaluation metadata are stored in timestamped folders alongside the model:
|
||||
`runs/your_run/final_model_evaluations/eval_<timestamp>/simulation.mp4`
|
||||
|
||||
For batch evaluation and cross-model comparison, see the **[Evaluation Guide](./evaluation.md)**.
|
||||
60
docs/api/tracking.md
Normal file
60
docs/api/tracking.md
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
# Tracking & Monitoring
|
||||
|
||||
This guide explains how to monitor your experiments using Weights & Biases (WandB) and TensorBoard.
|
||||
|
||||
## Weights & Biases (WandB)
|
||||
|
||||
WandB is used for online synchronization and visualization of training metrics.
|
||||
|
||||
### Authorization
|
||||
|
||||
Export your API key in your terminal to enable WandB synchronization:
|
||||
|
||||
```bash
|
||||
export WANDB_API_KEY=your_copied_api_key_here
|
||||
```
|
||||
|
||||
Alternatively, you can log in using the CLI:
|
||||
|
||||
```bash
|
||||
uv run wandb login
|
||||
```
|
||||
|
||||
### Enabling Tracking
|
||||
|
||||
To enable online sync during a training run, set `logging.track=true` on the command line:
|
||||
|
||||
```bash
|
||||
uv run python scripts/train.py logging.track=true
|
||||
```
|
||||
|
||||
You can also configure your project and entity:
|
||||
|
||||
```bash
|
||||
uv run python scripts/train.py \
|
||||
logging.track=true \
|
||||
logging.wandb_project_name="MyProject" \
|
||||
logging.wandb_entity="my-team"
|
||||
```
|
||||
|
||||
These can also be set in your configuration YAML file under the `logging` key.
|
||||
|
||||
## Local Monitoring with TensorBoard
|
||||
|
||||
All runs are recorded locally in the `runs/` directory (or the directory specified in `experiment.base_run_dir`). You can view scalars and other metrics with TensorBoard:
|
||||
|
||||
```bash
|
||||
tensorboard --logdir runs/
|
||||
```
|
||||
|
||||
Access the interface at `http://localhost:6006`.
|
||||
|
||||
### CLI Exploration Tool
|
||||
|
||||
For quick diagnostics or to export data to CSV without launching the full TensorBoard UI, you can use the `explore_tensorboard.py` script:
|
||||
|
||||
```bash
|
||||
uv run python scripts/analysis/explore_tensorboard.py runs/your_run_name/
|
||||
```
|
||||
|
||||
See the detailed description in [`/scripts/analysis/README.md`](../../scripts/analysis/README.md).
|
||||
55
docs/api/training.md
Normal file
55
docs/api/training.md
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
# Training Models
|
||||
|
||||
This guide covers how to configure and run training experiments for the Brittle Star project using Hydra-based configurations.
|
||||
|
||||
## Configuration
|
||||
|
||||
The project uses a modular configuration system powered by [Hydra](https://hydra.cc/). Instead of passing many command-line flags, you select and override configuration groups.
|
||||
|
||||
### Creating a Custom Experiment
|
||||
|
||||
1. **Create a new experiment file:**
|
||||
Create a file at `configs/experiment/my_experiment.yaml`. You can copy an existing one as a template:
|
||||
```bash
|
||||
cp configs/experiment/base.yaml configs/experiment/my_experiment.yaml
|
||||
```
|
||||
|
||||
2. **Edit `configs/experiment/my_experiment.yaml`** to set your experiment parameters:
|
||||
```yaml
|
||||
# @package _global_
|
||||
experiment:
|
||||
exp_name: "my_custom_run"
|
||||
seed: 42
|
||||
```
|
||||
|
||||
## Training Execution
|
||||
|
||||
To start a training run with the default settings defined in `configs/main_config.yaml`:
|
||||
|
||||
```bash
|
||||
uv run python scripts/train.py
|
||||
```
|
||||
|
||||
### Using a Custom Experiment Configuration
|
||||
|
||||
To run with your custom experiment file:
|
||||
|
||||
```bash
|
||||
uv run python scripts/train.py experiment=my_experiment
|
||||
```
|
||||
|
||||
```bash
|
||||
uv run python scripts/train.py ppo.learning_rate=0.001 ppo.num_envs=32 logging.track=true
|
||||
```
|
||||
|
||||
## Evaluation During Training
|
||||
|
||||
By default, the trainer saves checkpoints but does not evaluate them. To enable automatic headless evaluation of every saved checkpoint, set `evaluation.evaluate_checkpoints=true`:
|
||||
|
||||
```bash
|
||||
uv run python scripts/train.py evaluation.evaluate_checkpoints=true
|
||||
```
|
||||
|
||||
For more details on evaluation metrics and comparison tools, see [Evaluation](./evaluation.md).
|
||||
|
||||
For more details on tracking your experiments, see [Tracking & Monitoring](./tracking.md).
|
||||
|
|
@ -5,12 +5,26 @@ space (outputs). The control models map these observations directly to physical
|
|||
|
||||
**Inputs (state space)**
|
||||
|
||||
The observation space provides the agent with its current physical state and its objective.
|
||||
The observation space provides the agent with its current physical state and its navigational objective. With a
|
||||
decentralized control architecture in mind, we divide these inputs into global and local states.
|
||||
|
||||
- Joint positions: the current angles of all joints in the morphology.
|
||||
- Joint velocities: the current moving speed of the joints.
|
||||
- Goal vector: instad of just a scalar distance, the goal is represented asa a vector (distance and ange/direction) to
|
||||
the target.
|
||||
Global inputs, always broadcasted to all nodes:
|
||||
|
||||
- Vertical orientation/tilt: A single, simplified metric representing the tilt/vertical alignment of the agent's
|
||||
central body/disk, a.k.a. the deviation from the global Z-axis. Its value is derived from the environment's raw disk
|
||||
rotation 3D vector $[roll, pitch, yaw]$:
|
||||
$$
|
||||
tilt = sqrt(roll^2 + pitch^2)
|
||||
$$
|
||||
- Goal vector: A 2D unit vector representing the *egocentric* direction to the target. A value of $[1.0, 0.0]$
|
||||
indicates that the target is directly in front of the agent (angle 0).
|
||||
|
||||
Local inputs, routed directly to specific nodes:
|
||||
|
||||
- Joint positions: The current angles of all joints within the morphology.
|
||||
- Joint velocities: The current angular velocities of the joints.
|
||||
- Joint actuator forces: The physical forces currently exerted at each specific joint.
|
||||
- Segment contact: These values indicate whether each physical segment of the agent is currently touching the ground.
|
||||
|
||||
**Outputs (action space)**
|
||||
|
||||
|
|
@ -18,6 +32,16 @@ The action space defines how the agent interacts with the environment.
|
|||
|
||||
- Joint offsets: *absolute* target positions (offsets) for the joints, i.e. the exact angle the joint should move to.
|
||||
|
||||
## Normalization and Scaling
|
||||
|
||||
Both the input (observation) and output (action) spaces are rescaled to the range **$[-1, 1]$**.
|
||||
|
||||
For the input space, all raw physical values (angles, velocities, forces, distances) are normalized based on their
|
||||
defined physical bounds. If a value exceeds these bounds during simulation, it is clipped to the $[-1, 1]$ range.
|
||||
|
||||
For the output space, the neural network's tanh-activated outputs (which naturally fall in $[-1, 1]$) are linearly
|
||||
mapped to the physical joint limits defined in the robot's morphology.
|
||||
|
||||
## Rationale
|
||||
|
||||
When designing the state space, we must ask: *Could a human operator perform this task given only these inputs?*
|
||||
|
|
@ -26,15 +50,50 @@ When designing the state space, we must ask: *Could a human operator perform thi
|
|||
providing only the joint position is insufficient to determine the direction a limb is currently moving. By
|
||||
explicitly including joint velocities, the agent can immediately infer momentum and movement direction without
|
||||
needing to memorize past states.
|
||||
- Goal Vector (Distance + Angle): Providing only the scalar "distance to the goal" as an input is akin to blindfolding
|
||||
the robot and asking it to find a target by playing "hot or cold." By providing a full vector, the agent knows
|
||||
exactly where the target is relative to its current orientation, allowing for directed and efficient locomotion.
|
||||
- Absolute Joint Offsets: The physical Brittle Star robot relies on servo motors (if we were to build this simulated
|
||||
robot), which are inherently position-controlled devices. (Continuous rotation servos exist, but they are less
|
||||
commonly used for joints.) If our network outputted continuous torques (forces), a significant portion of the
|
||||
reinforcement learning process would be wasted on learning low-level PID control dynamics (i.e., how much force to
|
||||
apply to hold a position). Abstracting this away forces the learning algorithm to focus entirely on higher-level gait
|
||||
generation and locomotion.
|
||||
- Simplified vertical orientation: We drop the full 3D spatial rotation and angular velocity arrays in favor of a
|
||||
single vertical orientation metric (tilt). For a brittle star moving accross a flat plane, this metric is sufficient
|
||||
for the agent to sense if it is losing balance or flipping over.
|
||||
- Force representation: We strictly retain the joint actuator forces and drop the more generic actuator force. Forces
|
||||
that are explicitly tied to individual joints are significantly easier to route into decentralized, local limb nodes,
|
||||
which is necessary for our message-passing architecture.
|
||||
- Goal Vector (Distance + Angle): Providing only the scalar "distance to the goal" as an input is akin to blindfolding
|
||||
the robot and asking it to find a target by playing "hot or cold." By providing a full vector, the agent knows
|
||||
exactly where the target is relative to its current orientation, allowing for directed and efficient locomotion.
|
||||
|
||||
The goal representation is explicitly divided into a directional vector and a scalar distance. Keeping the direction
|
||||
as a normalized unit vector bounds the values to the $[-1, 1]$ range, which stabilizes neural network training.
|
||||
Providing only a scalar "distance to the goal" would force the agent to learning localized searching behaviors (e.g.
|
||||
random walks or spiraling) to deduce the direction, drastically increasing the difficulty of the learning task.
|
||||
|
||||
**NOTE:** We later dropped the "distance to vector", switching to only a direction as the input. Our reasoning is
|
||||
the agent should always move towards the goal (it should not learn to stop at the goal), which allows for this
|
||||
simplification that decreases the model input size.
|
||||
|
||||
The environment provides a raw `unit_xy_direction_to_target` (global), which we transform into a calculated
|
||||
`robot_direction_to_target` (egocentric) before passing it to the MLPs. This vector consists of the X and Y
|
||||
direction, where a value of $[1.0, 0.0]$ (mapping to an angle of $0$) means the robot is facing directly towards the
|
||||
target.
|
||||
- Contact sensors: Segment contact detects external ground interaction and is biologically vital for timing gait
|
||||
transitions.
|
||||
- Zero-Centered Rescaling ($[-1, 1]$): Using a zero-centered range is standard best practice for continuous control
|
||||
tasks. It provides several mathematical and physical advantages:
|
||||
- Improved Gradient Flow: Neural networks optimize faster when inputs are zero-centered. If all inputs were positive
|
||||
(e.g., $[0, 1]$), the gradients during backpropagation would be forced to the same sign, causing inefficient
|
||||
"zig-zag" weight updates.
|
||||
- Meaningful Neutral State: In robotics, $0.0$ naturally represents a resting state (zero velocity, centered
|
||||
position, no force). In a $[-1, 1]$ system, this physical rest maps to a neutral $0.0$ signal in the network.
|
||||
This also correctly communicaties a "neutral/dead" signal for amputated limbs that are padded with $0.0$ values.
|
||||
|
||||
Specifically, we do not include some available inputs:
|
||||
|
||||
- Global position: Absolute spatial coordinates can cause the agent to overfit to a specific coordinate frame or map,
|
||||
rather than learning general, adaptable locomotion strategies.
|
||||
|
||||
## Limitations and alternatives
|
||||
|
||||
|
|
@ -50,3 +109,56 @@ Alternative state and action formulations include:
|
|||
- Scalar Goal Distance: Giving the agent only the scalar distance to the target would force it to learn a localized
|
||||
searching behavior (e.g., spiraling or random walks) to determine the correct direction. While biologically plausible
|
||||
for simpler organisms following chemical gradients, it drastically increases the difficulty of the learning task.
|
||||
- **$[0, 1]$ Rescaling:** While some domains (like computer vision) use $[0, 1]$ scaling, it is generally avoided in
|
||||
robotics. Scaling to $[0, 1]$ would mean that a resting joint (velocity = 0) maps to an input of $0.5$. This
|
||||
constant positive bias forces the network to waste capacity learning to ignore or subtract this baseline signal just
|
||||
to stand still. Furthermore, it breaks the "dead signal" interpretation of zero-padding used for amputations.
|
||||
|
||||
## MuJoCo
|
||||
|
||||
This is what the filtered input vectors look like in MuJoCo, with $J$ joints and $S$ segments:
|
||||
|
||||
- `joint_position`: shape=(J,), dtype=float64
|
||||
- `joint_velocity`: shape=(J,), dtype=float64
|
||||
- `joint_actuator_force`: shape=(J,), dtype=float64
|
||||
- `segment_contact`: shape=(S,), dtype=float64
|
||||
- `robot_direction_to_target`: shape=(2,), dtype=float64, egocentric
|
||||
- `disk_z_tilt`: shape=(1,), dtype=float64, derived from `disk_rotation`
|
||||
|
||||
This brings the entire input space down to $3J + S + 4$ float64's, compared to $4J + S + 15$ float64's for the
|
||||
unfiltered inputs.
|
||||
|
||||
For reference, these are all the inputs that are available in the MuJoCo environment:
|
||||
|
||||
```
|
||||
obs keys: ['joint_position', 'joint_velocity', 'joint_actuator_force', 'actuator_force', 'disk_position', 'disk_rotation', 'disk_linear_velocity', 'disk_angular_velocity', 'tendon_position', 'tendon_velocity', 'segment_contact', 'unit_xy_direction_to_target', 'xy_distance_to_target']
|
||||
|
||||
raw observations dict:
|
||||
{'joint_position': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),
|
||||
'joint_velocity': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),
|
||||
'joint_actuator_force': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),
|
||||
'actuator_force': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),
|
||||
'disk_position': array([0. , 0. , 0.11]),
|
||||
'disk_rotation': (0.0, -0.0, 0.0),
|
||||
'disk_linear_velocity': array([0., 0., 0.]),
|
||||
'disk_angular_velocity': array([0., 0., 0.]),
|
||||
'tendon_position': array([], dtype=float64),
|
||||
'tendon_velocity': array([], dtype=float64),
|
||||
'segment_contact': array([0., 0., 0., 0., 0., 0.]),
|
||||
'unit_xy_direction_to_target': array([-0.95333378, -0.30191837]),
|
||||
'xy_distance_to_target': array([3.])}
|
||||
|
||||
(shapes)
|
||||
joint_position: shape=(12,), dtype=float64, size=12
|
||||
joint_velocity: shape=(12,), dtype=float64, size=12
|
||||
joint_actuator_force: shape=(12,), dtype=float64, size=12
|
||||
actuator_force: shape=(12,), dtype=float64, size=12
|
||||
disk_position: shape=(3,), dtype=float64, size=3
|
||||
disk_rotation: shape=(3,), dtype=float64, size=3
|
||||
disk_linear_velocity: shape=(3,), dtype=float64, size=3
|
||||
disk_angular_velocity: shape=(3,), dtype=float64, size=3
|
||||
tendon_position: shape=(0,), dtype=float64, size=0
|
||||
tendon_velocity: shape=(0,), dtype=float64, size=0
|
||||
segment_contact: shape=(6,), dtype=float64, size=6
|
||||
xy_distance_to_target: shape=(1,), dtype=float64, size=1
|
||||
```
|
||||
|
|
|
|||
Reference in a new issue