diff --git a/docs/design/communication.md b/docs/design/communication.md new file mode 100644 index 0000000..4a9cc82 --- /dev/null +++ b/docs/design/communication.md @@ -0,0 +1,43 @@ +# Communication scheme (Message Passing) + +Remember our research question: +> "What is the impact of different levels of controller modularity on learning speed, coordination, and fault tolerance +> (e.g. amputations) in brittle-star-like robots trained with Reinforcement Learning?" + +To test decentralized modularity (such as arm-level or segment-level controllers), the various modules *must* be able to +communicate with each other to achieve coordinated locomotion. This is accomplished through *message passing* in a Graph +Neural Network (GNN)-like architecture. Two prominent communication styles from the literature are N-step NerveNet (Wang +et al., 2018) and bottom-up top-down Shared Modular Policies (Huang et al., 2020). + +We have chosen to apply **one uniform communication style** across all modular architectures, specifically opting for +**N-step NerveNet**. + +## Rationale + +Initially, our idea was to equip arm-level controllers with NerveNet message passing and segment-level controllers with +SMP. However, we evaluated that this introduces a threat to the validity of our research question. If we observe +differences in performance, it would be impossible to determine whether the variance is caused by the *level of +modularity*, or by the difference in the message passing scheme. To purely compare modularity, the communication scheme +style must remain constant. + +Second, we decided that NerveNet is a better fit for our research. The morphology of our brittle star contains cycles at +the decentralized level (e.g., a ring of segments or arms around the body). NerveNet has proven to be robust for +arbitrary structures, including graphs with cycles. SMP inherently expects a tree structure for its bottom-up and +top-down pass. Applying SMP to a ring structure requires a workaround to break that cycle. + +## Limitations and alternatives + +Choosing NerveNet introduces a scalability issue as the morphology grows. In NerveNet, a message advances only one +segment or node per propagation step. When dealing with long arms (e.g., > 5 segments), this requires a large number of +propagation steps to transmit information from one tip of an arm to another. + +If we were to use SMP instead - which is possible - the inner states of nodes are shared across the entire graph in just +two passes. For very large or long morphologies, this would be much more scalable. + +By rejecting SMP, we accept that our model might learn slower or require more computational power for highly segmented, +extended morphologies. + +**References** + +- Wang, Tingwu, Renjie Liao, Jimmy Ba, and S. Fidler. ‘NerveNet: Learning Structured Policy with Graph Neural Networks’. Conference paper presented at International Conference on Learning Representations. 15 February 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613. +- Huang, Wenlong, Igor Mordatch, and Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 July 2020. https://doi.org/10.48550/arXiv.2007.04976. diff --git a/docs/design/controllers.md b/docs/design/controllers.md new file mode 100644 index 0000000..2dd59d9 --- /dev/null +++ b/docs/design/controllers.md @@ -0,0 +1,28 @@ +# Levels of modularity and topology + +The brittle star can be controlled at different levels. A monolithic controller processes all inputs and outputs at +once, whereas modular controllers divide the brains across the body, inspired by the biology of brittle stars. + +We define four architectures to compare: + +1. **Centralized, monolithic**: A single Multi Layer Perceptron per robot that receives all observations and outputs + all actions. +2. **Fully connected arm-level**: Each arm contains an MLP that processes the inputs for that arm, an MLP that processes + the communicated inner-states, and an MLP that outputs the actions for that arm. One policy for these MLPs is shared + across the arms. The controllers in each arm are connected to each other and form a fully connected graph. There is + no central disk, but the controllers are fully connected. +3. **Ring arm-level**: Identical setup to the fully connected arm-level, but the controllers are connected in a ring + structure. This setup is considered less centralized than the fully connected graph. +4. **Segment-level**: Each segment contains the three MLPs discussed above. The base segments, attached to the body, + form a ring structure, with the remaining segments attached as extended "strings". Segments can only communicate with + segments that are physically connected to it. + +## Rationale + +To fairly compare decentralized modularity against centralized control, the decentralized models should not be allowed +to contain a central organ acting as a bottleneck or coordinator. By removing the central disk in the decentralized +models and replacing it with a ring topology, we closely approximate the biological reality of the brittle star and test +a decentralized morphology. + +The fully connected graph functions as an intermediate step in between a fully centralized and a decentralized ring. We +use it to test whether our models scale to more complex structures. diff --git a/docs/design/learning_algorithm.md b/docs/design/learning_algorithm.md new file mode 100644 index 0000000..1b5c2f5 --- /dev/null +++ b/docs/design/learning_algorithm.md @@ -0,0 +1,30 @@ +# Reinforcement Learning Algorithm + +To control the *continuous* action space (the joints of the robot) based on sensor data, we require a reliable +Reinforcement Learning (RL) algorithm or optimization strategy. + +We have chosen **Proximal Policy Optimization (PPO)** (Schulman et al., 2017). + +## Rationale + +PPO is an on-policy algorithm known for its stability and robustness (safe training without excessive variance). More +importantly, it requires relatively little hyperparameter tuning compared to other algorithms. Since NerveNet was +successfully trained using PPO (Wang et al., 2018), selecting PPO significantly reduces the risk of convergence +failures. + +## Limitations and alternatives + +Alternative learning algorithms include: + +- **Twin Delayed DDPG (Fujimoto et al., 2018)**: TD3 is a strong off-policy alternative used in the SMP paper (Huang et + al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptations. However, this approach would + be more complex and error-prone than with PPO. +- **Evolution strategies (ES)**: Evolution strategies are useful for optimizing Central Pattern Generators (CPGs), e.g. + CMA-ES, OpenAI-ES. While this method is easier to distribute and parallelize, ES typically scales worse with + exceptionally large observation spaces compared to gradient-based RL methods like PPO. + +**References** + +- Fujimoto, Scott, Herke Hoof, and David Meger. ‘Addressing Function Approximation Error in Actor-Critic Methods’. Proceedings of the 35th International Conference on Machine Learning, 3 July 2018, 1587-96. https://proceedings.mlr.press/v80/fujimoto18a.html. +- Huang, Wenlong, Igor Mordatch, and Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 July 2020. https://doi.org/10.48550/arXiv.2007.04976. +- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. ‘Proximal Policy Optimization Algorithms’. arXiv:1707.06347. Preprint, arXiv, 28 August 2017. https://doi.org/10.48550/arXiv.1707.06347. diff --git a/docs/design/reward_function.md b/docs/design/reward_function.md new file mode 100644 index 0000000..1bf8725 --- /dev/null +++ b/docs/design/reward_function.md @@ -0,0 +1,26 @@ +# Reward function and observation space + +The robot needs to know whether its movements contribute to the ultimate goal of locomotion towards a target. Sensor +inputs must be distributed fairly to guarantee an objective comparison between different architectures. + +- The distance from the robot to the target and/or the light intensity are treated as global inputs. +- Positions and joints, which are normalized to floating-point values between 0 and 1, are considered local inputs. +- The reward function is centered around minimizing the distance to the goal or maximizing the movement towards the goal + within a finite number of timesteps $T$. + +## Rationale + +Using a light source (or a gradient) is biologically plausible for many simple organisms. By normalizing all signals +between 0 and 1, PPO training is highly stabilized. The timesteps must be finite to reset the environment in a timely +manner if the policy gets stuck in a local minimum. + +## Limitations and alternatives + +Providing global information to all individual decentralized segments can be considered biologically cheating or +practically infeasible once the robot would be physically built. Some sensory input cannot be put in each joint, for +example. + +The alternative is to provide the global input to the outermost segments of the arms, or a specific set of segments +assigned with this functionality. The network would then have to learn to propagate this signal throughout the body via +message passing. While biologically more accurate, this drastically complicates the learning process. We have written +this down as potential future research.