diff --git a/docs/design/actor-critic.md b/docs/design/actor-critic.md new file mode 100644 index 0000000..81fbc3e --- /dev/null +++ b/docs/design/actor-critic.md @@ -0,0 +1,68 @@ +# Actor-Critic Architecture + +To process observations into actions, our controllers utilize an Actor-Critic architecture. Because we use Proximal +Policy Optimization (PPO), the pipeline fundamentally requires separate networks for the policy (Actor) and the value +estimation (Critic). + +**Centralized Architecture (Baseline)** + +This pipeline treats the agent as a single entity and uses standard Proximal Policy Optimization (PPO). + +- Centralized Actor: Composed of two chained MLPs (Sensor $\rightarrow$ Motor) passing a hidden state between them. The + centralized sensor receives the concatenated global state vector of all limbs at once and processes it into a hidden + state. The centralized motor receives this hidden state and outputs the joint offsets for all actuators + simultaneously. This is mathematically equivalent to using one large MLP with hidden layers, but splitting makes the + implementation easier by allowing us to reuse the same components for the decentralized modules. +- Centralized Critic: Composed of two sequential MLPs (Feature Extractor $\rightarrow$ Critic). Because PPO evaluates + the state-value function, this network only receives the concatenated global state vector (no actions). It outputs a + single scalar estimating the expected future reward for the entire agent. + +Our policy and value networks use separate input networks/feature extractors as advised by the SEL3 course assistants and the blog. For continuous actions this should allow better learning at a small cost. + +**Decentralized Architecture** + +This pipeline utilizes the "Centralized Training with Decentralized Execution" principle, specifically the NerveNet-MLP +variant. + +- Decentralized Actor, split into three distinct models: + - Sensor: A local model at each node. It receives its local state plus the goal vector directly, processing them into + an initial hidden state. + - Propagation: Nodes synchronously compute and exchange messages with connected neighbors for $N$ steps to update + their hidden states. See [communication.md](./communication.md) for details. + - Motor: A local model uses its final updated hidden state to output the joint offset strictly for its own actuator. +- Centralized Critic: Composed of two sequential MLPs (Feature Extractor $\rightarrow$ Critic). During training, it + acts globally by taking the concatenated state vectors from all sensors to output a single, global state-value scalar + evaluating the entire agent's pose. + +To keep the implementation simple, we should use one critic per node in our architecture, but only a single, global +critic for all nodes at once, for the following reasons: + +1. Credit Assignment Problem (Ha, 2017): The MuJoCo simulator provides an overall reward based on the brittle star + movement progression, e.g. total distance travelled. Using an isolated critic for each node in the network would not + allow to determine which local action contributed to the global success. A global critic solves this by evaluating + the combined state of the agent at once. +2. Implementation simplicity: Building a second decentralized message-passing graph for the critic (NerveNet-2) would + require more coding. Using a standard MLP that concatenates all raw input vectors is much easier to program while + mathematically equivalent. + +## Implementation Details (Network Depth) + +Inspired by: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ + +The MLPs used in both pipelines are defined with specific hidden layer configurations to balance learning capability +and computational cost. As of right now, though this might change as we make progress in our experiments, we use: + +- Input Networks (Sensors & Feature Extractors): These networks map the raw state inputs to internal hidden states. + They are configured as standard dense networks with 2 hidden layers of 64 nodes each (`[64, 64]`) and utilize `tanh` + activation functions. +- Output Networks (Motors, Actors & Critics): The final output models are intentionally kept shallow. The Actor + directly projects the hidden state to a continuous action distribution (`mean` and `log_std`) using a single dense + output layer (zero hidden layers) initialized orthogonally. The Critic functions similarly, mapping the hidden + representation to a single scalar value. + +Note: For the continuous action distributions outputted by the Actor/Motor, we explicitly use mean and log_std as advised by previous research to maintain learning stability.ReferencesPPO Algorithm: Schulman et al. (2017), Proximal Policy Optimization Algorithms.Decentralized Message-Passing & NerveNet-MLP: Wang et al. (2018), NerveNet: Learning Structured Policy with Graph Neural Networks. + +**References** + +- Ha, D. (2017, October 29). A Visual Guide to Evolution Strategies. 大トロ ・ Machine Learning. https://blog.otoro.net/2017/10/29/visual-evolution-strategies/ +- Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. ‘Proximal Policy Optimization Algorithms’. arXiv:1707.06347. Preprint, arXiv, 28 August 2017. https://doi.org/10.48550/arXiv.1707.06347. diff --git a/docs/design/mlp_architecture.md b/docs/design/mlp_architecture.md deleted file mode 100644 index 93a6e51..0000000 --- a/docs/design/mlp_architecture.md +++ /dev/null @@ -1,24 +0,0 @@ -# Network Architecture (MLP Pipeline per Module) - -In the decentralized architectures (arm-level and segment-level), each controller/module follows the same **shared MLP-based pipeline** inspired by NerveNet-style message passing. The pipeline consists of 5 MLPs (4 in the case of centralized, with no messager): - -- **SENSOR**: Processes local observations for the actor branch. -- **FEATURE EXTRACTOR**: Processes local observations for the critic branch. -- **MESSAGER**: Processes incoming hidden states from neighboring modules (via the chosen communication scheme) and produces an aggregated hidden state. -- **ACTOR**: Takes the aggregated hidden state and outputs the action distribution (mean and log_std). -- **CRITIC**: Takes the aggregated hidden state and outputs a scalar value estimate. - -## MLP Pipeline - -![MLP Pipeline - Actor-Critic with Message Passing](../img/mlp_pipe.png "MLP neural network pipeline per module") - - -## Implementation Details Related To PPO - -Inspired by: https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ - -For starters we will execute our tests with simple models. Each MLP will have only 1 hidden layer. This will be expanded as needed. The exceptions are the input networks / feature extractors — they will be given 2 hidden layers and 64 nodes per layer as advised in the blog. This might change as we make progress in our experiments. - -Our policy and value networks use separate input networks / feature extractors as advised by the SEL3 course assistants and the blog. For continuous actions this should allow better learning at a small cost. - -We use mean and log_std to represent the action distribution, because it is advised by previous research for learning stability and other reasons. \ No newline at end of file