From a0d70947c16b696ae984e1ecbe95ce6403b80aba Mon Sep 17 00:00:00 2001 From: Tibo De Peuter Date: Sun, 15 Mar 2026 22:35:29 +0100 Subject: [PATCH] docs: Inputs and reward function --- docs/design/reward_function.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 docs/design/reward_function.md diff --git a/docs/design/reward_function.md b/docs/design/reward_function.md new file mode 100644 index 0000000..f18e1de --- /dev/null +++ b/docs/design/reward_function.md @@ -0,0 +1,26 @@ +# Reward function and observation space + +The robot needs to know whether its movements contribute to the ultimate goal of locomotion towards a target. Sensor +inputs must be distributed fairly to guarantee an objective comparison between different architectures. + +- The distance from the robot to the target and/or the light intensity are treated as global inputs. +- Positions and joints, normalized to floating-point values between 0 and 1 are considered local inputs. +- The reward function is centered around minimizing the distance to the goal or maximizing the movement towards the goal +within a finite number of timesteps $T$. + +## Rationale + +Using a light source (or a gradient) is biologically plausible for many simple organisms. By normalizing all signals +between 0 and 1, PPO training is highly stabilized. The timesteps must be finite to reset the environment in a timely +manner if the policy gets stuck in a local minimum. + +## Limitations and alternatives + +Providing global information to all individual decentralized segments can be considered biologically cheating or +practically infeasible once the robot would be physically built. Some sensory input cannot be put in each joint, for +example. + +The alternative is to provide the global input to the outermost segments of the arms, or a specific set of segments +assigned with this functionality. The network would then have to learn to propagate this signal throughout the body via +message passing. While biologically more accurate, this drastically complicates the learning process. We have written +this down as potential future research.