Critic Architecture Matters:
Dual vs. Unified Critics for Humanoid Loco-Manipulation

Mehmet Turan Yardımcı
Çukurova University, Computer Engineering · ORCID 0009-0004-8416-8368
ICRA 2026 Workshop on Reinforcement Learning for Imitation Learning (RL4IL)

Unitree G1 reaching toward a target during play evaluation (dual-critic policy, Isaac Lab).

Abstract

Multi-objective reinforcement learning for humanoid robots must coordinate locomotion and manipulation within a single policy. A natural design choice is whether to use a single (unified) critic that estimates the combined value of all objectives, or separate (dual) critics with disjoint reward signals. We compare the two on the Unitree G1 humanoid (23 active DoF, of which 17 are policy-controlled) in NVIDIA Isaac Lab, training loco-manipulation policies through sequential curricula that progress from stationary reaching to walking with variable-orientation targets. Under a matched compute budget, the dual-critic run reaches targets 3.5× faster (6.5 vs. 22.6 simulation steps), achieves 2× higher throughput (14.3 vs. 7.0 validated reaches per 1,000 steps), and attains a higher validated reach rate (65.2% vs. 53.8%) than the unified-critic run in a standardized evaluation. Adding five anti-gaming reward mechanisms on top of the dual critic yields no further improvement (60.9% vs. 65.2%). We report this as an efficiency gap between two trained policies rather than an isolated effect of the critic: the two runs also differ in curriculum schedule, arm action dimensionality and one locomotion reward weight, and each is a single seed. The results are nonetheless suggestive for the emerging paradigm of RL fine-tuning of imitation-learned policies, where a unified critic may suppress pre-trained arm behavior through competing locomotion gradients. We argue that critic architecture deserves explicit treatment as a design variable in multi-objective humanoid RL, and specify the single-variable ablation required to establish its causal contribution.

Results

Metric Unified critic (S6u) Dual critic (S6s) Dual + anti-gaming (S7)
Steps to target 22.6 6.5 5.8
Throughput (validated reaches / 1,000 steps) 7.0 14.3 13.0
Validated reach rate 53.8% 65.2% 60.9%

Standing evaluation: 3,000 steps, one environment, deterministic actions, seed 42. Adding five anti-gaming reward mechanisms on top of the dual critic (S7) does not improve on the architectural change alone.

Experimental caveats

These numbers come from a single pair of training runs, and those runs differ in more than the critic. What follows is what was not held constant.

  • The two runs did not train on the same curriculum. The unified-critic run used a 40-level ladder (reaching → orientation → gripper → height/payload); the dual-critic run used a 13-level one (standing → walking → fixed orientation → arbitrary orientation out to an 80° cone). These are different schedules, not long and short versions of one. When its checkpoint was saved the unified policy was at level 10 of 40 — standing still, 0.05 m tolerance, fixed palm-down orientation — while the dual policy had completed 12 of 12: walking at up to 0.6 m/s with an arbitrary commanded palm direction at 0.04 m tolerance.
  • Different locomotion reward. One of the fourteen locomotion weights differs: forward velocity tracking, 3.0 in the unified script against 5.0 in the dual one. This was active for the entire length of both runs.
  • Different arm action dimensionality. The unified policy's arm actor emits 12 values (5 arm + 7 finger) against the dual policy's 5. The seven finger outputs were sampled and entered the policy update, but never reached the robot — finger control activates at curriculum level 20 and the run ended at 10 — and they are discarded at evaluation. The confound is in exploration and policy entropy rather than in the task performed.
  • Single seed. The training scripts set no random seed at all, so each arm is one run with no variance estimate. The evaluation harness does seed itself, so the two policies are compared on matched target sequences.
  • Parallel environments. 2048 for the dual-critic run and 4096 for the anti-gaming run, both recorded. The unified run's launch flags were never saved; 2048 is inferred from a reach-counter ceiling argument, not read from a config.

What this means. The evaluation itself is apples-to-apples — one harness, one shared locomotion branch, matched target sequences — so the measured gap between these two checkpoints is real. The causal claim is not. With the curricula laid side by side, the simpler explanation for the gap is training progress: the two policies finished on tasks of very different difficulty, and the slower one had last been trained to stand still.

Critic architecture is therefore a hypothesis here, not an established cause. A controlled ablation — one curriculum, one action space, one reward set, only the critic swapped, across several seeds — is underway; these runs should be read as its starting point. Every training script now writes a full run_config.json so that any future pair can be checked field by field before being called a comparison.

Citation

@article{yardimci2026critic,
  title   = {Critic Architecture Matters: Dual vs. Unified Critics for
             Humanoid Loco-Manipulation},
  author  = {Yard{\i}mc{\i}, Mehmet Turan},
  journal = {arXiv preprint arXiv:2606.11891},
  year    = {2026},
  doi     = {10.48550/arXiv.2606.11891},
  url     = {https://github.com/mturan33/isaac-g1-ulc},
  note    = {ICRA 2026 Workshop on Reinforcement Learning for
             Imitation Learning (RL4IL)}
}