From ac0dd3891cbef84183c084935eebc13f2e396087 Mon Sep 17 00:00:00 2001 From: Tibo De Peuter Date: Sun, 15 Mar 2026 21:35:02 +0100 Subject: [PATCH 1/6] docs: Communication scheme --- docs/design/communication.md | 41 ++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 docs/design/communication.md diff --git a/docs/design/communication.md b/docs/design/communication.md new file mode 100644 index 0000000..8fd7615 --- /dev/null +++ b/docs/design/communication.md @@ -0,0 +1,41 @@ +# Communication scheme (Message Passing) + +Remember our research question: +> "What is the impact of different levels of controller modularity on learning speed, coordination, and fault tolerance +> (e.g. amputations) in brittle-star-like robots trained with Reinforcement Learning?" + +To test decentralized modularity (such as arm-level or segment-level controllers), the various modules *must* be able to +communicate with each other to achieve coordinated locomotion. This is accomplished through *message passing* in a Graph +Neural Network (GNN)-like architecture. Two prominent communication styles from the literature are N-step NerveNet (Wang +et al., 2018) and bottom-up top-down Shared Modular Policies (Huang et al., 2020). + +We have chosen to apply **one uniform communication style** across all modular architectures, specifically opting for +**N-step NerveNet**. + +## Rationale + +Initially, our idea was to equip arm-level controllers with NerveNet message passing and segment-level controllers with +SMP. However, we evaluated that this introduces a threat to the validity of our research question. If we observe +differences in performance, it would be impossible to determine whether the variance is caused by the *level of +modularity*, or by the difference in the message passing scheme. To purely compare modularity, the communication scheme +style must remain constant. + +Second, we decided that NerveNet is a better fit for our research. The morphology of our brittle star contains cycles at +the decentralized level (e.g., a ring of segments or arms around the body). NerveNet has proven to be robust for +arbitrary structures, including graphs with cycles. SMP inherently expects a tree structure for its bottom-up and +top-down pass. Applying SMP to a ring structure requires a workaround to break that cycle. + +## Limitations and alternatives + +Choosing NerveNet introduces a scalability issue as the morphology grows. In NerveNet, a message advances only one +segment or node per propagation step. When dealing with long arms (e.g., > 5 segments), this requires a large number of +propagation steps to transmit information from one tip of an arm to another. + +If we would alternatively use SMP - which is possible - the inner states of nodes are shared across the entire graph in +just two passes. For very large or long morphologies, this would be much more scalable. + +By rejecting SMP, we accept that our model might learn slower or require more computational power for highly segmented, +extended morphologies. + +- Wang, Tingwu, Renjie Liao, Jimmy Ba, en S. Fidler. ‘NerveNet: Learning Structured Policy with Graph Neural Networks’. Conference paper presented bij International Conference on Learning Representations. 15 februari 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613. +- Huang, Wenlong, Igor Mordatch, en Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 juli 2020. https://doi.org/10.48550/arXiv.2007.04976. From 29759ef4f4bae9048b84baa77f27acfc90e497fa Mon Sep 17 00:00:00 2001 From: Tibo De Peuter Date: Sun, 15 Mar 2026 22:09:41 +0100 Subject: [PATCH 2/6] docs: Modularity --- docs/design/controllers.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 docs/design/controllers.md diff --git a/docs/design/controllers.md b/docs/design/controllers.md new file mode 100644 index 0000000..3405c46 --- /dev/null +++ b/docs/design/controllers.md @@ -0,0 +1,28 @@ +# Levels of modularity and topology + +The brittle star can be controlled at different levels. A monolithic controller processes all inputs and outputs at +once, whereas modular controllers divide the brains across the body, inspired by the biology of brittle stars. + +We define four architectures to compare: + +1. **Centralized, monolithic**: A single Multi Layer Perceptron per robot that receives all observations and outputs +all actions. +2. **Fully connected arm-level**: Each arm contains an MLP that processes the inputs for that arm, an MLP that processes +the communicated inner-states, and an MLP that outputs the actions for that arm. One policy for these MLPs is shared +across the arms. The controllers in each arm are connected to each other and form a fully connected graph. There is no +central disk, but the controllers are fully connected. +3. **Ring arm-level**: Identical setup to the fully connected arm-level, but the controllers are connected in a ring +structure. This setup is considered less centralised than the fully connected graph. +4. **Segment-level**: Each segment contains the three MLPs discussed above. The base segments, attached to the body, +form a ring structure, with the remaining segments attached as extended "strings". Segments can only communicate with +segments that are physically connected to it. + +## Rationale + +To fairly compare decentralized modularity against centralized control, the decentralized models should not be allowed +to contain a central organ acting as a bottleneck or coordinator. By removing the central disk in the decentralized +models and replacing it with a ring topology, we closely approximate the biological reality of the brittle star and test +a decentralized morphology. + +The fully connected graph functions as an intermediate step in between a fully centralised and a decentralised ring. We +use it to test if the scaling of our models to more complex structures. From 59f78a67086b8d63517ed021be5d6056eb172dff Mon Sep 17 00:00:00 2001 From: Tibo De Peuter Date: Sun, 15 Mar 2026 22:26:40 +0100 Subject: [PATCH 3/6] docs: Learning algorithm --- docs/design/communication.md | 2 ++ docs/design/learning_algorithm.md | 30 ++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+) create mode 100644 docs/design/learning_algorithm.md diff --git a/docs/design/communication.md b/docs/design/communication.md index 8fd7615..24b3faa 100644 --- a/docs/design/communication.md +++ b/docs/design/communication.md @@ -37,5 +37,7 @@ just two passes. For very large or long morphologies, this would be much more sc By rejecting SMP, we accept that our model might learn slower or require more computational power for highly segmented, extended morphologies. +**References** + - Wang, Tingwu, Renjie Liao, Jimmy Ba, en S. Fidler. ‘NerveNet: Learning Structured Policy with Graph Neural Networks’. Conference paper presented bij International Conference on Learning Representations. 15 februari 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613. - Huang, Wenlong, Igor Mordatch, en Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 juli 2020. https://doi.org/10.48550/arXiv.2007.04976. diff --git a/docs/design/learning_algorithm.md b/docs/design/learning_algorithm.md new file mode 100644 index 0000000..e28c10e --- /dev/null +++ b/docs/design/learning_algorithm.md @@ -0,0 +1,30 @@ +# Reinforcement Learning Algorithm + +To control the *continuous* action space (the joints of the robot) based on sensor data, we require a reliable +Reinforcement Learning (RL) algorithm or optimization strategy. + +We have chosen **Proximal Policy Optimization (PPO)** (Schulman et al., 2017). + +## Rationale + +PPO is an on-policy algorithm known for its stability and robustness (safe training without excessive variance). More +importantly, it requires relatively little hyperparameter tuning compared to other algorithms. Since NerveNet was +successfully trained using PPO (Wang et al., 2018), selecting PPO significantly reduces the risk of convergence +failures. + +## Limitations and alternatives + +Alternative learning algorithms include: + +- **Twin Delayed DDPG (Fujimoto et al., 2018)**: TD3 is a strong off-policy alternative used in the SMP paper (Huang et +al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptions. However, this approach would be +more complex and error-prone than with PPO. +- **Evolution strategies (ES)**: Evolution strategies are useful for optimizing Central Pattern Generators (CPGs), e.g. +CMA-ES, OpenAI-ES. While this method is easier to distribute and parallelize, ES typically scales worse with +exceptionally large observation spaces compared to gradient-based RL methods like PPO. + +**References** + +- Fujimoto, Scott, Herke Hoof, en David Meger. ‘Addressing Function Approximation Error in Actor-Critic Methods’. Proceedings of the 35th International Conference on Machine Learning, 3 juli 2018, 1587-96. https://proceedings.mlr.press/v80/fujimoto18a.html. +- Huang, Wenlong, Igor Mordatch, en Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 juli 2020. https://doi.org/10.48550/arXiv.2007.04976. +- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, en Oleg Klimov. ‘Proximal Policy Optimization Algorithms’. arXiv:1707.06347. Preprint, arXiv, 28 augustus 2017. https://doi.org/10.48550/arXiv.1707.06347. From a0d70947c16b696ae984e1ecbe95ce6403b80aba Mon Sep 17 00:00:00 2001 From: Tibo De Peuter Date: Sun, 15 Mar 2026 22:35:29 +0100 Subject: [PATCH 4/6] docs: Inputs and reward function --- docs/design/reward_function.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 docs/design/reward_function.md diff --git a/docs/design/reward_function.md b/docs/design/reward_function.md new file mode 100644 index 0000000..f18e1de --- /dev/null +++ b/docs/design/reward_function.md @@ -0,0 +1,26 @@ +# Reward function and observation space + +The robot needs to know whether its movements contribute to the ultimate goal of locomotion towards a target. Sensor +inputs must be distributed fairly to guarantee an objective comparison between different architectures. + +- The distance from the robot to the target and/or the light intensity are treated as global inputs. +- Positions and joints, normalized to floating-point values between 0 and 1 are considered local inputs. +- The reward function is centered around minimizing the distance to the goal or maximizing the movement towards the goal +within a finite number of timesteps $T$. + +## Rationale + +Using a light source (or a gradient) is biologically plausible for many simple organisms. By normalizing all signals +between 0 and 1, PPO training is highly stabilized. The timesteps must be finite to reset the environment in a timely +manner if the policy gets stuck in a local minimum. + +## Limitations and alternatives + +Providing global information to all individual decentralized segments can be considered biologically cheating or +practically infeasible once the robot would be physically built. Some sensory input cannot be put in each joint, for +example. + +The alternative is to provide the global input to the outermost segments of the arms, or a specific set of segments +assigned with this functionality. The network would then have to learn to propagate this signal throughout the body via +message passing. While biologically more accurate, this drastically complicates the learning process. We have written +this down as potential future research. From bc9419ca53d007c3d0ffc8c13f565b6ca9ebbddf Mon Sep 17 00:00:00 2001 From: Tibo De Peuter Date: Sun, 15 Mar 2026 23:00:30 +0100 Subject: [PATCH 5/6] Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/design/communication.md | 4 ++-- docs/design/controllers.md | 8 ++++---- docs/design/learning_algorithm.md | 8 ++++---- docs/design/reward_function.md | 4 ++-- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/design/communication.md b/docs/design/communication.md index 24b3faa..1fe546b 100644 --- a/docs/design/communication.md +++ b/docs/design/communication.md @@ -39,5 +39,5 @@ extended morphologies. **References** -- Wang, Tingwu, Renjie Liao, Jimmy Ba, en S. Fidler. ‘NerveNet: Learning Structured Policy with Graph Neural Networks’. Conference paper presented bij International Conference on Learning Representations. 15 februari 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613. -- Huang, Wenlong, Igor Mordatch, en Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 juli 2020. https://doi.org/10.48550/arXiv.2007.04976. +- Wang, Tingwu, Renjie Liao, Jimmy Ba, and S. Fidler. ‘NerveNet: Learning Structured Policy with Graph Neural Networks’. Conference paper presented at International Conference on Learning Representations. 15 February 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613. +- Huang, Wenlong, Igor Mordatch, and Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 July 2020. https://doi.org/10.48550/arXiv.2007.04976. diff --git a/docs/design/controllers.md b/docs/design/controllers.md index 3405c46..4688908 100644 --- a/docs/design/controllers.md +++ b/docs/design/controllers.md @@ -12,7 +12,7 @@ the communicated inner-states, and an MLP that outputs the actions for that arm. across the arms. The controllers in each arm are connected to each other and form a fully connected graph. There is no central disk, but the controllers are fully connected. 3. **Ring arm-level**: Identical setup to the fully connected arm-level, but the controllers are connected in a ring -structure. This setup is considered less centralised than the fully connected graph. +structure. This setup is considered less centralized than the fully connected graph. 4. **Segment-level**: Each segment contains the three MLPs discussed above. The base segments, attached to the body, form a ring structure, with the remaining segments attached as extended "strings". Segments can only communicate with segments that are physically connected to it. @@ -22,7 +22,7 @@ segments that are physically connected to it. To fairly compare decentralized modularity against centralized control, the decentralized models should not be allowed to contain a central organ acting as a bottleneck or coordinator. By removing the central disk in the decentralized models and replacing it with a ring topology, we closely approximate the biological reality of the brittle star and test -a decentralized morphology. +a decentralized morphology. -The fully connected graph functions as an intermediate step in between a fully centralised and a decentralised ring. We -use it to test if the scaling of our models to more complex structures. +The fully connected graph functions as an intermediate step in between a fully centralized and a decentralized ring. We +use it to test whether our models scale to more complex structures. diff --git a/docs/design/learning_algorithm.md b/docs/design/learning_algorithm.md index e28c10e..6365e1e 100644 --- a/docs/design/learning_algorithm.md +++ b/docs/design/learning_algorithm.md @@ -17,7 +17,7 @@ failures. Alternative learning algorithms include: - **Twin Delayed DDPG (Fujimoto et al., 2018)**: TD3 is a strong off-policy alternative used in the SMP paper (Huang et -al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptions. However, this approach would be +al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptations. However, this approach would be more complex and error-prone than with PPO. - **Evolution strategies (ES)**: Evolution strategies are useful for optimizing Central Pattern Generators (CPGs), e.g. CMA-ES, OpenAI-ES. While this method is easier to distribute and parallelize, ES typically scales worse with @@ -25,6 +25,6 @@ exceptionally large observation spaces compared to gradient-based RL methods lik **References** -- Fujimoto, Scott, Herke Hoof, en David Meger. ‘Addressing Function Approximation Error in Actor-Critic Methods’. Proceedings of the 35th International Conference on Machine Learning, 3 juli 2018, 1587-96. https://proceedings.mlr.press/v80/fujimoto18a.html. -- Huang, Wenlong, Igor Mordatch, en Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 juli 2020. https://doi.org/10.48550/arXiv.2007.04976. -- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, en Oleg Klimov. ‘Proximal Policy Optimization Algorithms’. arXiv:1707.06347. Preprint, arXiv, 28 augustus 2017. https://doi.org/10.48550/arXiv.1707.06347. +- Fujimoto, Scott, Herke Hoof, and David Meger. ‘Addressing Function Approximation Error in Actor-Critic Methods’. Proceedings of the 35th International Conference on Machine Learning, 3 July 2018, 1587-96. https://proceedings.mlr.press/v80/fujimoto18a.html. +- Huang, Wenlong, Igor Mordatch, and Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 July 2020. https://doi.org/10.48550/arXiv.2007.04976. +- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. ‘Proximal Policy Optimization Algorithms’. arXiv:1707.06347. Preprint, arXiv, 28 August 2017. https://doi.org/10.48550/arXiv.1707.06347. diff --git a/docs/design/reward_function.md b/docs/design/reward_function.md index f18e1de..1bf8725 100644 --- a/docs/design/reward_function.md +++ b/docs/design/reward_function.md @@ -4,9 +4,9 @@ The robot needs to know whether its movements contribute to the ultimate goal of inputs must be distributed fairly to guarantee an objective comparison between different architectures. - The distance from the robot to the target and/or the light intensity are treated as global inputs. -- Positions and joints, normalized to floating-point values between 0 and 1 are considered local inputs. +- Positions and joints, which are normalized to floating-point values between 0 and 1, are considered local inputs. - The reward function is centered around minimizing the distance to the goal or maximizing the movement towards the goal -within a finite number of timesteps $T$. + within a finite number of timesteps $T$. ## Rationale From 94f505920cf071906b030297940306cac76cbdcf Mon Sep 17 00:00:00 2001 From: Tibo De Peuter Date: Sun, 15 Mar 2026 23:03:09 +0100 Subject: [PATCH 6/6] style: Fix Markdown indentation --- docs/design/communication.md | 4 ++-- docs/design/controllers.md | 14 +++++++------- docs/design/learning_algorithm.md | 8 ++++---- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/design/communication.md b/docs/design/communication.md index 1fe546b..4a9cc82 100644 --- a/docs/design/communication.md +++ b/docs/design/communication.md @@ -31,8 +31,8 @@ Choosing NerveNet introduces a scalability issue as the morphology grows. In Ner segment or node per propagation step. When dealing with long arms (e.g., > 5 segments), this requires a large number of propagation steps to transmit information from one tip of an arm to another. -If we would alternatively use SMP - which is possible - the inner states of nodes are shared across the entire graph in -just two passes. For very large or long morphologies, this would be much more scalable. +If we were to use SMP instead - which is possible - the inner states of nodes are shared across the entire graph in just +two passes. For very large or long morphologies, this would be much more scalable. By rejecting SMP, we accept that our model might learn slower or require more computational power for highly segmented, extended morphologies. diff --git a/docs/design/controllers.md b/docs/design/controllers.md index 4688908..2dd59d9 100644 --- a/docs/design/controllers.md +++ b/docs/design/controllers.md @@ -6,16 +6,16 @@ once, whereas modular controllers divide the brains across the body, inspired by We define four architectures to compare: 1. **Centralized, monolithic**: A single Multi Layer Perceptron per robot that receives all observations and outputs -all actions. + all actions. 2. **Fully connected arm-level**: Each arm contains an MLP that processes the inputs for that arm, an MLP that processes -the communicated inner-states, and an MLP that outputs the actions for that arm. One policy for these MLPs is shared -across the arms. The controllers in each arm are connected to each other and form a fully connected graph. There is no -central disk, but the controllers are fully connected. + the communicated inner-states, and an MLP that outputs the actions for that arm. One policy for these MLPs is shared + across the arms. The controllers in each arm are connected to each other and form a fully connected graph. There is + no central disk, but the controllers are fully connected. 3. **Ring arm-level**: Identical setup to the fully connected arm-level, but the controllers are connected in a ring -structure. This setup is considered less centralized than the fully connected graph. + structure. This setup is considered less centralized than the fully connected graph. 4. **Segment-level**: Each segment contains the three MLPs discussed above. The base segments, attached to the body, -form a ring structure, with the remaining segments attached as extended "strings". Segments can only communicate with -segments that are physically connected to it. + form a ring structure, with the remaining segments attached as extended "strings". Segments can only communicate with + segments that are physically connected to it. ## Rationale diff --git a/docs/design/learning_algorithm.md b/docs/design/learning_algorithm.md index 6365e1e..1b5c2f5 100644 --- a/docs/design/learning_algorithm.md +++ b/docs/design/learning_algorithm.md @@ -17,11 +17,11 @@ failures. Alternative learning algorithms include: - **Twin Delayed DDPG (Fujimoto et al., 2018)**: TD3 is a strong off-policy alternative used in the SMP paper (Huang et -al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptations. However, this approach would be -more complex and error-prone than with PPO. + al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptations. However, this approach would + be more complex and error-prone than with PPO. - **Evolution strategies (ES)**: Evolution strategies are useful for optimizing Central Pattern Generators (CPGs), e.g. -CMA-ES, OpenAI-ES. While this method is easier to distribute and parallelize, ES typically scales worse with -exceptionally large observation spaces compared to gradient-based RL methods like PPO. + CMA-ES, OpenAI-ES. While this method is easier to distribute and parallelize, ES typically scales worse with + exceptionally large observation spaces compared to gradient-based RL methods like PPO. **References**