Merge pull request #2 from SELab-3-2026/docs/research-design

docs: Formally document research design

docs/design/communication.md

@@ -0,0 +1,43 @@
+# Communication scheme (Message Passing)
+
+Remember our research question:
+> "What is the impact of different levels of controller modularity on learning speed, coordination, and fault tolerance
+> (e.g. amputations) in brittle-star-like robots trained with Reinforcement Learning?"
+
+To test decentralized modularity (such as arm-level or segment-level controllers), the various modules *must* be able to
+communicate with each other to achieve coordinated locomotion. This is accomplished through *message passing* in a Graph
+Neural Network (GNN)-like architecture. Two prominent communication styles from the literature are N-step NerveNet (Wang
+et al., 2018) and bottom-up top-down Shared Modular Policies (Huang et al., 2020).
+
+We have chosen to apply **one uniform communication style** across all modular architectures, specifically opting for
+**N-step NerveNet**.
+
+## Rationale
+
+Initially, our idea was to equip arm-level controllers with NerveNet message passing and segment-level controllers with
+SMP. However, we evaluated that this introduces a threat to the validity of our research question. If we observe
+differences in performance, it would be impossible to determine whether the variance is caused by the *level of
+modularity*, or by the difference in the message passing scheme. To purely compare modularity, the communication scheme
+style must remain constant.
+
+Second, we decided that NerveNet is a better fit for our research. The morphology of our brittle star contains cycles at
+the decentralized level (e.g., a ring of segments or arms around the body). NerveNet has proven to be robust for
+arbitrary structures, including graphs with cycles. SMP inherently expects a tree structure for its bottom-up and
+top-down pass. Applying SMP to a ring structure requires a workaround to break that cycle.
+
+## Limitations and alternatives
+
+Choosing NerveNet introduces a scalability issue as the morphology grows. In NerveNet, a message advances only one
+segment or node per propagation step. When dealing with long arms (e.g., > 5 segments), this requires a large number of
+propagation steps to transmit information from one tip of an arm to another.
+
+If we were to use SMP instead - which is possible - the inner states of nodes are shared across the entire graph in just
+two passes. For very large or long morphologies, this would be much more scalable.
+
+By rejecting SMP, we accept that our model might learn slower or require more computational power for highly segmented,
+extended morphologies.
+
+**References**
+
+- Wang, Tingwu, Renjie Liao, Jimmy Ba, and S. Fidler. ‘NerveNet: Learning Structured Policy with Graph Neural Networks’. Conference paper presented at International Conference on Learning Representations. 15 February 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613.
+- Huang, Wenlong, Igor Mordatch, and Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 July 2020. https://doi.org/10.48550/arXiv.2007.04976.

docs/design/controllers.md

@@ -0,0 +1,28 @@
+# Levels of modularity and topology
+
+The brittle star can be controlled at different levels. A monolithic controller processes all inputs and outputs at
+once, whereas modular controllers divide the brains across the body, inspired by the biology of brittle stars.
+
+We define four architectures to compare:
+
+1. **Centralized, monolithic**: A single Multi Layer Perceptron per robot that receives all observations and outputs
+   all actions.
+2. **Fully connected arm-level**: Each arm contains an MLP that processes the inputs for that arm, an MLP that processes
+   the communicated inner-states, and an MLP that outputs the actions for that arm. One policy for these MLPs is shared
+   across the arms. The controllers in each arm are connected to each other and form a fully connected graph. There is
+   no central disk, but the controllers are fully connected.
+3. **Ring arm-level**: Identical setup to the fully connected arm-level, but the controllers are connected in a ring
+   structure. This setup is considered less centralized than the fully connected graph.
+4. **Segment-level**: Each segment contains the three MLPs discussed above. The base segments, attached to the body,
+   form a ring structure, with the remaining segments attached as extended "strings". Segments can only communicate with
+   segments that are physically connected to it.
+
+## Rationale
+
+To fairly compare decentralized modularity against centralized control, the decentralized models should not be allowed
+to contain a central organ acting as a bottleneck or coordinator. By removing the central disk in the decentralized
+models and replacing it with a ring topology, we closely approximate the biological reality of the brittle star and test
+a decentralized morphology.
+
+The fully connected graph functions as an intermediate step in between a fully centralized and a decentralized ring. We
+use it to test whether our models scale to more complex structures.

docs/design/learning_algorithm.md

@@ -0,0 +1,30 @@
+# Reinforcement Learning Algorithm
+
+To control the *continuous* action space (the joints of the robot) based on sensor data, we require a reliable
+Reinforcement Learning (RL) algorithm or optimization strategy.
+
+We have chosen **Proximal Policy Optimization (PPO)** (Schulman et al., 2017).
+
+## Rationale
+
+PPO is an on-policy algorithm known for its stability and robustness (safe training without excessive variance). More
+importantly, it requires relatively little hyperparameter tuning compared to other algorithms. Since NerveNet was
+successfully trained using PPO (Wang et al., 2018), selecting PPO significantly reduces the risk of convergence
+failures.
+
+## Limitations and alternatives
+
+Alternative learning algorithms include:
+
+- **Twin Delayed DDPG (Fujimoto et al., 2018)**: TD3 is a strong off-policy alternative used in the SMP paper (Huang et
+  al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptations. However, this approach would
+  be more complex and error-prone than with PPO.
+- **Evolution strategies (ES)**: Evolution strategies are useful for optimizing Central Pattern Generators (CPGs), e.g.
+  CMA-ES, OpenAI-ES. While this method is easier to distribute and parallelize, ES typically scales worse with
+  exceptionally large observation spaces compared to gradient-based RL methods like PPO.
+
+**References**
+
+- Fujimoto, Scott, Herke Hoof, and David Meger. ‘Addressing Function Approximation Error in Actor-Critic Methods’. Proceedings of the 35th International Conference on Machine Learning, 3 July 2018, 1587-96. https://proceedings.mlr.press/v80/fujimoto18a.html.
+- Huang, Wenlong, Igor Mordatch, and Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 July 2020. https://doi.org/10.48550/arXiv.2007.04976.
+- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. ‘Proximal Policy Optimization Algorithms’. arXiv:1707.06347. Preprint, arXiv, 28 August 2017. https://doi.org/10.48550/arXiv.1707.06347.

docs/design/reward_function.md

@@ -0,0 +1,26 @@
+# Reward function and observation space
+
+The robot needs to know whether its movements contribute to the ultimate goal of locomotion towards a target. Sensor
+inputs must be distributed fairly to guarantee an objective comparison between different architectures.
+
+- The distance from the robot to the target and/or the light intensity are treated as global inputs.
+- Positions and joints, which are normalized to floating-point values between 0 and 1, are considered local inputs.
+- The reward function is centered around minimizing the distance to the goal or maximizing the movement towards the goal
+  within a finite number of timesteps $T$.
+
+## Rationale
+
+Using a light source (or a gradient) is biologically plausible for many simple organisms. By normalizing all signals
+between 0 and 1, PPO training is highly stabilized. The timesteps must be finite to reset the environment in a timely
+manner if the policy gets stuck in a local minimum.
+
+## Limitations and alternatives
+
+Providing global information to all individual decentralized segments can be considered biologically cheating or
+practically infeasible once the robot would be physically built. Some sensory input cannot be put in each joint, for
+example.
+
+The alternative is to provide the global input to the outermost segments of the arms, or a specific set of segments
+assigned with this functionality. The network would then have to learn to propagate this signal throughout the body via
+message passing. While biologically more accurate, this drastically complicates the learning process. We have written
+this down as potential future research.