docs: detailed actor-critic pipelines

docs/design/actor-critic.md

@@ -0,0 +1,68 @@
+# Actor-Critic Architecture
+
+To process observations into actions, our controllers utilize an Actor-Critic architecture. Because we use Proximal
+Policy Optimization (PPO), the pipeline fundamentally requires separate networks for the policy (Actor) and the value
+estimation (Critic).
+
+**Centralized Architecture (Baseline)**
+
+This pipeline treats the agent as a single entity and uses standard Proximal Policy Optimization (PPO).
+
+- Centralized Actor: Composed of two chained MLPs (Sensor $\rightarrow$ Motor) passing a hidden state between them. The
+  centralized sensor receives the concatenated global state vector of all limbs at once and processes it into a hidden
+  state. The centralized motor receives this hidden state and outputs the joint offsets for all actuators
+  simultaneously. This is mathematically equivalent to using one large MLP with hidden layers, but splitting makes the
+  implementation easier by allowing us to reuse the same components for the decentralized modules.
+- Centralized Critic: Composed of two sequential MLPs (Feature Extractor $\rightarrow$ Critic). Because PPO evaluates
+  the state-value function, this network only receives the concatenated global state vector (no actions). It outputs a
+  single scalar estimating the expected future reward for the entire agent.
+
+Our policy and value networks use separate input networks/feature extractors as advised by the SEL3 course assistants and the blog. For continuous actions this should allow better learning at a small cost.
+
+**Decentralized Architecture**
+
+This pipeline utilizes the "Centralized Training with Decentralized Execution" principle, specifically the NerveNet-MLP
+variant.
+
+- Decentralized Actor, split into three distinct models:
+  - Sensor: A local model at each node. It receives its local state plus the goal vector directly, processing them into
+    an initial hidden state.
+  - Propagation: Nodes synchronously compute and exchange messages with connected neighbors for $N$ steps to update
+    their hidden states. See [communication.md](./communication.md) for details.
+  - Motor: A local model uses its final updated hidden state to output the joint offset strictly for its own actuator.
+- Centralized Critic: Composed of two sequential MLPs (Feature Extractor $\rightarrow$ Critic). During training, it
+  acts globally by taking the concatenated state vectors from all sensors to output a single, global state-value scalar
+  evaluating the entire agent's pose.
+
+To keep the implementation simple, we should use one critic per node in our architecture, but only a single, global
+critic for all nodes at once, for the following reasons:
+
+1. Credit Assignment Problem (Ha, 2017): The MuJoCo simulator provides an overall reward based on the brittle star
+  movement progression, e.g. total distance travelled. Using an isolated critic for each node in the network would not
+  allow to determine which local action contributed to the global success. A global critic solves this by evaluating
+  the combined state of the agent at once.
+2. Implementation simplicity: Building a second decentralized message-passing graph for the critic (NerveNet-2) would
+  require more coding. Using a standard MLP that concatenates all raw input vectors is much easier to program while
+  mathematically equivalent.
+
+## Implementation Details (Network Depth)
+
+Inspired by: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/
+
+The MLPs used in both pipelines are defined with specific hidden layer configurations to balance learning capability
+and computational cost. As of right now, though this might change as we make progress in our experiments, we use:
+
+- Input Networks (Sensors & Feature Extractors): These networks map the raw state inputs to internal hidden states.
+  They are configured as standard dense networks with 2 hidden layers of 64 nodes each (`[64, 64]`) and utilize `tanh`
+  activation functions.
+- Output Networks (Motors, Actors & Critics): The final output models are intentionally kept shallow. The Actor
+  directly projects the hidden state to a continuous action distribution (`mean` and `log_std`) using a single dense
+  output layer (zero hidden layers) initialized orthogonally. The Critic functions similarly, mapping the hidden
+  representation to a single scalar value.
+
+Note: For the continuous action distributions outputted by the Actor/Motor, we explicitly use mean and log_std as advised by previous research to maintain learning stability.ReferencesPPO Algorithm: Schulman et al. (2017), Proximal Policy Optimization Algorithms.Decentralized Message-Passing & NerveNet-MLP: Wang et al. (2018), NerveNet: Learning Structured Policy with Graph Neural Networks.
+
+**References**
+
+- Ha, D. (2017, October 29). A Visual Guide to Evolution Strategies. 大トロ ・ Machine Learning. https://blog.otoro.net/2017/10/29/visual-evolution-strategies/
+- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. ‘Proximal Policy Optimization Algorithms’. arXiv:1707.06347. Preprint, arXiv, 28 August 2017. https://doi.org/10.48550/arXiv.1707.06347.

docs/design/mlp_architecture.md

@@ -1,24 +0,0 @@
-# Network Architecture (MLP Pipeline per Module)
-
-In the decentralized architectures (arm-level and segment-level), each controller/module follows the same **shared MLP-based pipeline** inspired by NerveNet-style message passing. The pipeline consists of 5 MLPs (4 in the case of centralized, with no messager):
-
-- **SENSOR**: Processes local observations for the actor branch.
-- **FEATURE EXTRACTOR**: Processes local observations for the critic branch.
-- **MESSAGER**: Processes incoming hidden states from neighboring modules (via the chosen communication scheme) and produces an aggregated hidden state.
-- **ACTOR**: Takes the aggregated hidden state and outputs the action distribution (mean and log_std).
-- **CRITIC**: Takes the aggregated hidden state and outputs a scalar value estimate.
-
-## MLP Pipeline
-
-![MLP Pipeline - Actor-Critic with Message Passing](../img/mlp_pipe.png "MLP neural network pipeline per module")
-
-
-## Implementation Details Related To PPO
-
-Inspired by: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/
-
-For starters we will execute our tests with simple models. Each MLP will have only 1 hidden layer. This will be expanded as needed. The exceptions are the input networks / feature extractors — they will be given 2 hidden layers and 64 nodes per layer as advised in the blog. This might change as we make progress in our experiments.
-
-Our policy and value networks use separate input networks / feature extractors as advised by the SEL3 course assistants and the blog. For continuous actions this should allow better learning at a small cost.
-
-We use mean and log_std to represent the action distribution, because it is advised by previous research for learning stability and other reasons.