diff --git a/docs/design/reward_function.md b/docs/design/reward_function.md new file mode 100644 index 0000000..f18e1de --- /dev/null +++ b/docs/design/reward_function.md @@ -0,0 +1,26 @@ +# Reward function and observation space + +The robot needs to know whether its movements contribute to the ultimate goal of locomotion towards a target. Sensor +inputs must be distributed fairly to guarantee an objective comparison between different architectures. + +- The distance from the robot to the target and/or the light intensity are treated as global inputs. +- Positions and joints, normalized to floating-point values between 0 and 1 are considered local inputs. +- The reward function is centered around minimizing the distance to the goal or maximizing the movement towards the goal +within a finite number of timesteps $T$. + +## Rationale + +Using a light source (or a gradient) is biologically plausible for many simple organisms. By normalizing all signals +between 0 and 1, PPO training is highly stabilized. The timesteps must be finite to reset the environment in a timely +manner if the policy gets stuck in a local minimum. + +## Limitations and alternatives + +Providing global information to all individual decentralized segments can be considered biologically cheating or +practically infeasible once the robot would be physically built. Some sensory input cannot be put in each joint, for +example. + +The alternative is to provide the global input to the outermost segments of the arms, or a specific set of segments +assigned with this functionality. The network would then have to learn to propagate this signal throughout the body via +message passing. While biologically more accurate, this drastically complicates the learning process. We have written +this down as potential future research.