diff --git a/docs/design/communication.md b/docs/design/communication.md index 1fe546b..4a9cc82 100644 --- a/docs/design/communication.md +++ b/docs/design/communication.md @@ -31,8 +31,8 @@ Choosing NerveNet introduces a scalability issue as the morphology grows. In Ner segment or node per propagation step. When dealing with long arms (e.g., > 5 segments), this requires a large number of propagation steps to transmit information from one tip of an arm to another. -If we would alternatively use SMP - which is possible - the inner states of nodes are shared across the entire graph in -just two passes. For very large or long morphologies, this would be much more scalable. +If we were to use SMP instead - which is possible - the inner states of nodes are shared across the entire graph in just +two passes. For very large or long morphologies, this would be much more scalable. By rejecting SMP, we accept that our model might learn slower or require more computational power for highly segmented, extended morphologies. diff --git a/docs/design/controllers.md b/docs/design/controllers.md index 4688908..2dd59d9 100644 --- a/docs/design/controllers.md +++ b/docs/design/controllers.md @@ -6,16 +6,16 @@ once, whereas modular controllers divide the brains across the body, inspired by We define four architectures to compare: 1. **Centralized, monolithic**: A single Multi Layer Perceptron per robot that receives all observations and outputs -all actions. + all actions. 2. **Fully connected arm-level**: Each arm contains an MLP that processes the inputs for that arm, an MLP that processes -the communicated inner-states, and an MLP that outputs the actions for that arm. One policy for these MLPs is shared -across the arms. The controllers in each arm are connected to each other and form a fully connected graph. There is no -central disk, but the controllers are fully connected. + the communicated inner-states, and an MLP that outputs the actions for that arm. One policy for these MLPs is shared + across the arms. The controllers in each arm are connected to each other and form a fully connected graph. There is + no central disk, but the controllers are fully connected. 3. **Ring arm-level**: Identical setup to the fully connected arm-level, but the controllers are connected in a ring -structure. This setup is considered less centralized than the fully connected graph. + structure. This setup is considered less centralized than the fully connected graph. 4. **Segment-level**: Each segment contains the three MLPs discussed above. The base segments, attached to the body, -form a ring structure, with the remaining segments attached as extended "strings". Segments can only communicate with -segments that are physically connected to it. + form a ring structure, with the remaining segments attached as extended "strings". Segments can only communicate with + segments that are physically connected to it. ## Rationale diff --git a/docs/design/learning_algorithm.md b/docs/design/learning_algorithm.md index 6365e1e..1b5c2f5 100644 --- a/docs/design/learning_algorithm.md +++ b/docs/design/learning_algorithm.md @@ -17,11 +17,11 @@ failures. Alternative learning algorithms include: - **Twin Delayed DDPG (Fujimoto et al., 2018)**: TD3 is a strong off-policy alternative used in the SMP paper (Huang et -al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptations. However, this approach would be -more complex and error-prone than with PPO. + al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptations. However, this approach would + be more complex and error-prone than with PPO. - **Evolution strategies (ES)**: Evolution strategies are useful for optimizing Central Pattern Generators (CPGs), e.g. -CMA-ES, OpenAI-ES. While this method is easier to distribute and parallelize, ES typically scales worse with -exceptionally large observation spaces compared to gradient-based RL methods like PPO. + CMA-ES, OpenAI-ES. While this method is easier to distribute and parallelize, ES typically scales worse with + exceptionally large observation spaces compared to gradient-based RL methods like PPO. **References**