docs: Learning algorithm

docs/design/communication.md

@@ -37,5 +37,7 @@ just two passes. For very large or long morphologies, this would be much more sc
 By rejecting SMP, we accept that our model might learn slower or require more computational power for highly segmented,
 extended morphologies.
 
+**References**
+
 - Wang, Tingwu, Renjie Liao, Jimmy Ba, en S. Fidler. ‘NerveNet: Learning Structured Policy with Graph Neural Networks’. Conference paper presented bij International Conference on Learning Representations. 15 februari 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613.
 - Huang, Wenlong, Igor Mordatch, en Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 juli 2020. https://doi.org/10.48550/arXiv.2007.04976.

docs/design/learning_algorithm.md

@@ -0,0 +1,30 @@
+# Reinforcement Learning Algorithm
+
+To control the *continuous* action space (the joints of the robot) based on sensor data, we require a reliable
+Reinforcement Learning (RL) algorithm or optimization strategy.
+
+We have chosen **Proximal Policy Optimization (PPO)** (Schulman et al., 2017).
+
+## Rationale
+
+PPO is an on-policy algorithm known for its stability and robustness (safe training without excessive variance). More
+importantly, it requires relatively little hyperparameter tuning compared to other algorithms. Since NerveNet was
+successfully trained using PPO (Wang et al., 2018), selecting PPO significantly reduces the risk of convergence
+failures.
+
+## Limitations and alternatives
+
+Alternative learning algorithms include:
+
+- **Twin Delayed DDPG (Fujimoto et al., 2018)**: TD3 is a strong off-policy alternative used in the SMP paper (Huang et
+al., 2020). It is highly sample-efficient and reportedly excels at zero-shot adaptions. However, this approach would be
+more complex and error-prone than with PPO.
+- **Evolution strategies (ES)**: Evolution strategies are useful for optimizing Central Pattern Generators (CPGs), e.g.
+CMA-ES, OpenAI-ES. While this method is easier to distribute and parallelize, ES typically scales worse with
+exceptionally large observation spaces compared to gradient-based RL methods like PPO.
+
+**References**
+
+- Fujimoto, Scott, Herke Hoof, en David Meger. ‘Addressing Function Approximation Error in Actor-Critic Methods’. Proceedings of the 35th International Conference on Machine Learning, 3 juli 2018, 1587-96. https://proceedings.mlr.press/v80/fujimoto18a.html.
+- Huang, Wenlong, Igor Mordatch, en Deepak Pathak. ‘One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control’. arXiv:2007.04976. Preprint, arXiv, 9 juli 2020. https://doi.org/10.48550/arXiv.2007.04976.
+- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, en Oleg Klimov. ‘Proximal Policy Optimization Algorithms’. arXiv:1707.06347. Preprint, arXiv, 28 augustus 2017. https://doi.org/10.48550/arXiv.1707.06347.