From bc9419ca53d007c3d0ffc8c13f565b6ca9ebbddf Mon Sep 17 00:00:00 2001 From: Tibo De Peuter Date: Sun, 15 Mar 2026 23:00:30 +0100 Subject: [PATCH] Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/design/communication.md | 4 ++-- docs/design/controllers.md | 8 ++++---- docs/design/learning_algorithm.md | 8 ++++---- docs/design/reward_function.md | 4 ++-- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/design/communication.md b/docs/design/communication.md index 24b3faa..1fe546b 100644 --- a/docs/design/communication.md +++ b/docs/design/communication.md @@ -39,5 +39,5 @@ extended morphologies. **References** -- Wang, Tingwu, Renjie Liao, Jimmy Ba, en S. Fidler. ‘NerveNet: Learning Structured Policy with Graph Neural Networks’. Conference paper presented bij International Conference on Learning Representations. 15 februari 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613. -- Huang, Wenlong, Igor Mordatch, en Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 juli 2020. https://doi.org/10.48550/arXiv.2007.04976. +- Wang, Tingwu, Renjie Liao, Jimmy Ba, and S. Fidler. ‘NerveNet: Learning Structured Policy with Graph Neural Networks’. Conference paper presented at International Conference on Learning Representations. 15 February 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613. +- Huang, Wenlong, Igor Mordatch, and Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 July 2020. https://doi.org/10.48550/arXiv.2007.04976. diff --git a/docs/design/controllers.md b/docs/design/controllers.md index 3405c46..4688908 100644 --- a/docs/design/controllers.md +++ b/docs/design/controllers.md @@ -12,7 +12,7 @@ the communicated inner-states, and an MLP that outputs the actions for that arm. across the arms. The controllers in each arm are connected to each other and form a fully connected graph. There is no central disk, but the controllers are fully connected. 3. **Ring arm-level**: Identical setup to the fully connected arm-level, but the controllers are connected in a ring -structure. This setup is considered less centralised than the fully connected graph. +structure. This setup is considered less centralized than the fully connected graph. 4. **Segment-level**: Each segment contains the three MLPs discussed above. The base segments, attached to the body, form a ring structure, with the remaining segments attached as extended "strings". Segments can only communicate with segments that are physically connected to it. @@ -22,7 +22,7 @@ segments that are physically connected to it. To fairly compare decentralized modularity against centralized control, the decentralized models should not be allowed to contain a central organ acting as a bottleneck or coordinator. By removing the central disk in the decentralized models and replacing it with a ring topology, we closely approximate the biological reality of the brittle star and test -a decentralized morphology. +a decentralized morphology. -The fully connected graph functions as an intermediate step in between a fully centralised and a decentralised ring. We -use it to test if the scaling of our models to more complex structures. +The fully connected graph functions as an intermediate step in between a fully centralized and a decentralized ring. We +use it to test whether our models scale to more complex structures. diff --git a/docs/design/learning_algorithm.md b/docs/design/learning_algorithm.md index e28c10e..6365e1e 100644 --- a/docs/design/learning_algorithm.md +++ b/docs/design/learning_algorithm.md @@ -17,7 +17,7 @@ failures. Alternative learning algorithms include: - **Twin Delayed DDPG (Fujimoto et al., 2018)**: TD3 is a strong off-policy alternative used in the SMP paper (Huang et -al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptions. However, this approach would be +al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptations. However, this approach would be more complex and error-prone than with PPO. - **Evolution strategies (ES)**: Evolution strategies are useful for optimizing Central Pattern Generators (CPGs), e.g. CMA-ES, OpenAI-ES. While this method is easier to distribute and parallelize, ES typically scales worse with @@ -25,6 +25,6 @@ exceptionally large observation spaces compared to gradient-based RL methods lik **References** -- Fujimoto, Scott, Herke Hoof, en David Meger. ‘Addressing Function Approximation Error in Actor-Critic Methods’. Proceedings of the 35th International Conference on Machine Learning, 3 juli 2018, 1587-96. https://proceedings.mlr.press/v80/fujimoto18a.html. -- Huang, Wenlong, Igor Mordatch, en Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 juli 2020. https://doi.org/10.48550/arXiv.2007.04976. -- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, en Oleg Klimov. ‘Proximal Policy Optimization Algorithms’. arXiv:1707.06347. Preprint, arXiv, 28 augustus 2017. https://doi.org/10.48550/arXiv.1707.06347. +- Fujimoto, Scott, Herke Hoof, and David Meger. ‘Addressing Function Approximation Error in Actor-Critic Methods’. Proceedings of the 35th International Conference on Machine Learning, 3 July 2018, 1587-96. https://proceedings.mlr.press/v80/fujimoto18a.html. +- Huang, Wenlong, Igor Mordatch, and Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 July 2020. https://doi.org/10.48550/arXiv.2007.04976. +- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. ‘Proximal Policy Optimization Algorithms’. arXiv:1707.06347. Preprint, arXiv, 28 August 2017. https://doi.org/10.48550/arXiv.1707.06347. diff --git a/docs/design/reward_function.md b/docs/design/reward_function.md index f18e1de..1bf8725 100644 --- a/docs/design/reward_function.md +++ b/docs/design/reward_function.md @@ -4,9 +4,9 @@ The robot needs to know whether its movements contribute to the ultimate goal of inputs must be distributed fairly to guarantee an objective comparison between different architectures. - The distance from the robot to the target and/or the light intensity are treated as global inputs. -- Positions and joints, normalized to floating-point values between 0 and 1 are considered local inputs. +- Positions and joints, which are normalized to floating-point values between 0 and 1, are considered local inputs. - The reward function is centered around minimizing the distance to the goal or maximizing the movement towards the goal -within a finite number of timesteps $T$. + within a finite number of timesteps $T$. ## Rationale