protomotions.agents.peft.prior_amp_config module#
Configuration for discrete-prior PEFT with AMP style rewards.
- class protomotions.agents.peft.prior_amp_config.DiscretePriorPEFTRLFTAMPModelConfig(
- _target_='protomotions.agents.peft.prior_amp_model.DiscretePriorPEFTRLFTAMPModel',
- in_keys=<factory>,
- out_keys=<factory>,
- actor=<factory>,
- actor_optimizer=<factory>,
- critic=None,
- critic_optimizer=<factory>,
- discriminator=<factory>,
- discriminator_optimizer=<factory>,
- disc_critic=<factory>,
- disc_critic_optimizer=<factory>,
Bases:
DiscretePriorPEFTRLFTModelConfigDiscrete-prior PEFT RLFT model plus AMP discriminator modules.
- Attributes:
in_keys: Input keys. out_keys: Output keys. critic: Task critic config. RLFT requires this to be set.
- discriminator: DiscriminatorConfig#
- discriminator_optimizer: OptimizerConfig#
- disc_critic: ModuleContainerConfig#
- disc_critic_optimizer: OptimizerConfig#
- __init__(
- _target_='protomotions.agents.peft.prior_amp_model.DiscretePriorPEFTRLFTAMPModel',
- in_keys=<factory>,
- out_keys=<factory>,
- actor=<factory>,
- actor_optimizer=<factory>,
- critic=None,
- critic_optimizer=<factory>,
- discriminator=<factory>,
- discriminator_optimizer=<factory>,
- disc_critic=<factory>,
- disc_critic_optimizer=<factory>,
- class protomotions.agents.peft.prior_amp_config.DiscretePriorPEFTRLFTAMPAgentConfig(
- batch_size,
- training_max_steps,
- _target_='protomotions.agents.peft.prior_amp_agent.DiscretePriorPEFTRLFTAMPAgent',
- model=<factory>,
- num_steps=32,
- gradient_clip_val=25.0,
- fail_on_bad_grads=False,
- check_grad_mag=True,
- gamma=0.99,
- bounds_loss_coef=0.0,
- task_reward_w=1.0,
- num_mini_epochs=2,
- training_early_termination=None,
- save_epoch_checkpoint_every=1000,
- save_last_checkpoint_every=10,
- save_inference_checkpoint=True,
- max_episode_length_manager=None,
- evaluator=<factory>,
- normalize_rewards=True,
- normalized_reward_clamp_value=5.0,
- reward_norm_ema_decay=None,
- tau=0.95,
- e_clip=0.2,
- clip_critic_loss=True,
- actor_clip_frac_threshold=0.6,
- entropy_coef=0.0,
- l2c2=<factory>,
- adaptive_lr=<factory>,
- advantage_normalization=<factory>,
- pretrained_modules=<factory>,
- target_kl=None,
- amp_parameters=<factory>,
- reference_obs_components=<factory>,
Bases:
DiscretePriorPEFTRLFTAgentConfigDiscrete-prior PEFT RLFT agent augmented with AMP rewards.
- Attributes:
batch_size: Training batch size. training_max_steps: Maximum training steps. num_steps: Environment steps per update. fail_on_bad_grads: Fail on NaN/Inf gradients. check_grad_mag: Log gradient magnitude. gamma: Discount factor. bounds_loss_coef: Action bounds loss. 0 for tanh outputs. task_reward_w: Task reward weight. training_early_termination: Stop early at this step. None=disabled. save_epoch_checkpoint_every: Save epoch_xxx.ckpt every N epochs. save_last_checkpoint_every: Save last.ckpt every K epochs. max_episode_length_manager: Episode length curriculum. evaluator: Evaluation config. normalize_rewards: Normalize rewards. normalized_reward_clamp_value: Clamp normalized rewards to [-val, val]. reward_norm_ema_decay: EMA decay for reward normalization (None = Welford). Set to e.g. 0.99 to track non-stationary reward distributions. tau: GAE lambda for advantage estimation. e_clip: PPO clipping parameter epsilon. clip_critic_loss: Clip critic loss similar to actor. actor_clip_frac_threshold: Skip actor update if clip_frac > threshold (e.g., 0.5). entropy_coef: Entropy bonus coefficient added to PPO actor loss. Keep 0.0 for SFT-warmed PEFT unless deliberately exploring. l2c2: L2C2 settings. adaptive_lr: Adaptive learning rate settings. advantage_normalization: Advantage normalization settings. pretrained_modules: Frozen modules keyed by name. PEFT expects a whole prior model under ‘prior’. target_kl: If set, skip actor updates once minibatch actor/kl exceeds target_kl * 1.5 for the current rollout update.
- amp_parameters: AMPParametersConfig#
- __init__(
- batch_size,
- training_max_steps,
- _target_='protomotions.agents.peft.prior_amp_agent.DiscretePriorPEFTRLFTAMPAgent',
- model=<factory>,
- num_steps=32,
- gradient_clip_val=25.0,
- fail_on_bad_grads=False,
- check_grad_mag=True,
- gamma=0.99,
- bounds_loss_coef=0.0,
- task_reward_w=1.0,
- num_mini_epochs=2,
- training_early_termination=None,
- save_epoch_checkpoint_every=1000,
- save_last_checkpoint_every=10,
- save_inference_checkpoint=True,
- max_episode_length_manager=None,
- evaluator=<factory>,
- normalize_rewards=True,
- normalized_reward_clamp_value=5.0,
- reward_norm_ema_decay=None,
- tau=0.95,
- e_clip=0.2,
- clip_critic_loss=True,
- actor_clip_frac_threshold=0.6,
- entropy_coef=0.0,
- l2c2=<factory>,
- adaptive_lr=<factory>,
- advantage_normalization=<factory>,
- pretrained_modules=<factory>,
- target_kl=None,
- amp_parameters=<factory>,
- reference_obs_components=<factory>,