docs: add actor/critic pipeline figures

docs/design/actor-critic.md

@@ -19,6 +19,29 @@ This pipeline treats the agent as a single entity and uses standard Proximal Pol
 
 Our policy and value networks use separate input networks/feature extractors as advised by the SEL3 course assistants and the blog. For continuous actions this should allow better learning at a small cost.
 
+```mermaid
+%%{ init: { 'flowchart': {'defaultRenderer': 'elk' } } }%%
+graph TD
+    Obs([Global Observation])
+
+    Sens[Sensor]
+    Act[Motor]
+    OutAct([Action Distribution<br/>mean, log_std])
+
+    Feat[Feature extractor]
+    Crit[Critic]
+    OutCrit([Value Estimate<br/>scalar])
+
+    Obs --> Sens
+    Obs --> Feat
+
+    Sens -->|"Hidden state"| Act
+    Feat -->|"Hidden state"| Crit
+
+    Act --> OutAct
+    Crit --> OutCrit
+```
+
 **Decentralized Architecture**
 
 This pipeline utilizes the "Centralized Training with Decentralized Execution" principle, specifically the NerveNet-MLP
@@ -27,7 +50,7 @@ variant.
 - Decentralized Actor, split into three distinct models:
   - Sensor: A local model at each node. It receives its local state plus the goal vector directly, processing them into
     an initial hidden state.
-  - Propagation: Nodes synchronously compute and exchange messages with connected neighbors for $N$ steps to update
+  - Propagator: Nodes synchronously compute and exchange messages with connected neighbors for $N$ steps to update
     their hidden states. See [communication.md](./communication.md) for details.
   - Motor: A local model uses its final updated hidden state to output the joint offset strictly for its own actuator.
 - Centralized Critic: Composed of two sequential MLPs (Feature Extractor $\rightarrow$ Critic). During training, it
@@ -45,6 +68,37 @@ critic for all nodes at once, for the following reasons:
   require more coding. Using a standard MLP that concatenates all raw input vectors is much easier to program while
   mathematically equivalent.
 
+```mermaid
+%%{ init: { 'flowchart': {'defaultRenderer': 'elk' } } }%%
+graph TD
+    Obs([Local Observation])
+
+    Sens[Sensor]
+    Prop[Propagator]
+    Feat[Feature extractor]
+
+    Mot[Motor]
+    Crit[Critic]
+
+    OutMot([Action Distribution<br/>mean, log_std])
+    OutCrit([Value Estimate<br/>scalar])
+
+    Obs --> Sens
+    Sens -->|"Hidden state"| Prop
+    Obs --> Feat
+
+    Prop -->|"Hidden state"| Mot
+
+
+    Feat -->|"Hidden state"| Crit
+
+    Mot --> OutMot
+    Crit --> OutCrit
+
+    Prop -.->|"message passing"|Prop
+
+```
+
 ## Implementation Details (Network Depth)
 
 Inspired by: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/
@@ -60,9 +114,11 @@ and computational cost. As of right now, though this might change as we make pro
   output layer (zero hidden layers) initialized orthogonally. The Critic functions similarly, mapping the hidden
   representation to a single scalar value.
 
-Note: For the continuous action distributions outputted by the Actor/Motor, we explicitly use mean and log_std as advised by previous research to maintain learning stability.ReferencesPPO Algorithm: Schulman et al. (2017), Proximal Policy Optimization Algorithms.Decentralized Message-Passing & NerveNet-MLP: Wang et al. (2018), NerveNet: Learning Structured Policy with Graph Neural Networks.
+Note: For the continuous action distributions outputted by the Motor, we explicitly use `mean` and `log_std` as advised
+by previous research to maintain learning stability.
 
 **References**
 
 - Ha, D. (2017, October 29). A Visual Guide to Evolution Strategies. 大トロ ・ Machine Learning. https://blog.otoro.net/2017/10/29/visual-evolution-strategies/
 - Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. ‘Proximal Policy Optimization Algorithms’. arXiv:1707.06347. Preprint, arXiv, 28 August 2017. https://doi.org/10.48550/arXiv.1707.06347.
+- Wang, Tingwu, Renjie Liao, Jimmy Ba, and S. Fidler. ‘NerveNet: Learning Structured Policy with Graph Neural Networks’. Conference paper presented at International Conference on Learning Representations. 15 February 2018. https://www.semanticscholar.org/paper/NerveNet:-Learning-Structured-Policy-with-Graph-Wang-Liao/249408527106d7595d45dd761dd53c83e5a02613.