protomotions.agents.ppo.model module

PPO model implementation with actor-critic architecture.

This module implements the neural network models for Proximal Policy Optimization. The actor outputs a Gaussian policy distribution, and the critic estimates state values.

Key Classes:

• PPOActor: Policy network with Gaussian action distribution

• PPOModel: Complete actor-critic model for PPO

class protomotions.agents.ppo.model.PPOActor(*args, **kwargs)

Bases: ProtoMotionsTensorDictModule

PPO policy network (actor).

Self-contained policy that computes distribution parameters, samples actions, and computes log probabilities all in a single forward pass.

Parameters:

config (PPOActorConfig) – Actor configuration including network architecture and initial log std.

Variables:

• logstd – Log standard deviation parameter (typically fixed during training).

• mu – Neural network that outputs action means.

• in_keys – List of input keys from mu model.

• out_keys – List of output keys (action, mean_action, neglogp).

__init__(config)

experience_buffer_keys()

Keys produced during rollout that the experience buffer must store.

forward_deployable(tensordict, log_internals=False)

Compute the deterministic policy without sampling exploration noise.

property deployable_in_keys: list

The mean path reads the mu model’s inputs, not the actor’s full set.

forward(tensordict, log_internals=False)

Compute the mean, sample an action and return its negative log probability.

class protomotions.agents.ppo.model.PPOModel(*args, **kwargs)

Bases: BaseModel

Complete PPO model with actor and critic networks.

Pure forward function that computes all model outputs in TensorDict. The forward pass adds action distribution parameters and value estimates.

Parameters:

config (PPOModelConfig) – Model configuration specifying actor and critic architectures.

Variables:

• _actor – Policy network.

• _critic – Value network.

config: PPOModelConfig

experience_buffer_keys()

Keys produced during rollout that the experience buffer must store.

__init__(config)

forward(tensordict, log_internals=False)

Forward pass through actor and critic.

Computes all outputs: - action: Sampled action - mean_action: Deterministic action (mean) - neglogp: Negative log probability of sampled action - value: State value estimate

Parameters:

tensordict (MockTensorDict) – TensorDict containing observations.

Returns:

TensorDict with all model outputs added.

Return type:

MockTensorDict

forward_deployable(tensordict)

The deployable graph is the actor’s mean path; the critic is not exported.

property deployable_in_keys: list

The input keys the deployable graph reads; in_keys by default.

property deployable_action_key: str

The output key the host drives the robot from.