diff --git a/docs/README.md b/docs/README.md index 8c8d659..ac56e31 100644 --- a/docs/README.md +++ b/docs/README.md @@ -2,10 +2,11 @@ ## Design & architecture ([`/design`](./design/)) -- [Controllers](./design/controllers.md): Macroscopig brain toplogy, centralized, arm-level, segment-level. -- [MLP architecture](./design/mlp_architecture.md): Module wiring, state/action spaces, and actor-critic pipeline. - [Communication](./design/communication.md): Message propagation, Nerve-Net style. +- [Controllers](./design/controllers.md): Macroscopig brain toplogy, centralized, arm-level, segment-level. +- [Input/output](./design/input_action_spaces.md): Description of the model's input and output. - [Learning algorithm](./design/learning_algorithm.md): RL techniques, i.e. PPO. +- [MLP architecture](./design/mlp_architecture.md): Module wiring, state/action spaces, and actor-critic pipeline. - [Reward function](./design/learning_algorithm.md): Goals, fitness tracking, and reward structures. ## API reference ([`/api`](./api/)) diff --git a/docs/design/input_action_spaces.md b/docs/design/input_action_spaces.md new file mode 100644 index 0000000..7860e16 --- /dev/null +++ b/docs/design/input_action_spaces.md @@ -0,0 +1,52 @@ +# Input (state) and output (action) spaces + +To effectively learn locomotion and navigation, the agent requires a well-defined observation space (inputs) and action +space (outputs). The control models map these observations directly to physical movements. + +**Inputs (state space)** + +The observation space provides the agent with its current physical state and its objective. + +- Joint positions: the current angles of all joints in the morphology. +- Joint velocities: the current moving speed of the joints. +- Goal vector: instad of just a scalar distance, the goal is represented asa a vector (distance and ange/direction) to + the target. + +**Outputs (action space)** + +The action space defines how the agent interacts with the environment. + +- Joint offsets: *absolute* target positions (offsets) for the joints, i.e. the exact angle the joint should move to. + +## Rationale + +When designing the state space, we must ask: *Could a human operator perform this task given only these inputs?* + +- Inclusion of Joint Velocities: Because our control models do not inherently possess memory of previous timesteps, + providing only the joint position is insufficient to determine the direction a limb is currently moving. By + explicitly including joint velocities, the agent can immediately infer momentum and movement direction without + needing to memorize past states. +- Goal Vector (Distance + Angle): Providing only the scalar "distance to the goal" as an input is akin to blindfolding + the robot and asking it to find a target by playing "hot or cold." By providing a full vector, the agent knows + exactly where the target is relative to its current orientation, allowing for directed and efficient locomotion. +- Absolute Joint Offsets: The physical Brittle Star robot relies on servo motors (if we were to build this simulated + robot), which are inherently position-controlled devices. (Continuous rotation servos exist, but they are less + commonly used for joints.) If our network outputted continuous torques (forces), a significant portion of the + reinforcement learning process would be wasted on learning low-level PID control dynamics (i.e., how much force to + apply to hold a position). Abstracting this away forces the learning algorithm to focus entirely on higher-level gait + generation and locomotion. + +## Limitations and alternatives + +Alternative state and action formulations include: + +- Torque-based continuous control: In many continuous control tasks (like standard MuJoCo benchmarks), actions + represent continuous torques applied to joints. While this provides more granular, low-level physical control, it + heavily complicates training and does not align well with the physical reality of servo-driven hardware. +- Recurrent Neural Networks (RNNs) / Frame Stacking: Instead of explicitly passing velocities in the state space, the + network could infer momentum by observing a history of past states. Using RNNs or frame stacking allows the agent to + build an internal memory of movement. However, this significantly increases architectural complexity and training + time compared to explicitly providing the velocity data. +- Scalar Goal Distance: Giving the agent only the scalar distance to the target would force it to learn a localized + searching behavior (e.g., spiraling or random walks) to determine the correct direction. While biologically plausible + for simpler organisms following chemical gradients, it drastically increases the difficulty of the learning task.