Critic Architecture Matters:
Dual vs. Unified Critics for Humanoid Loco-Manipulation

Mehmet Turan Yardımcı
Çukurova University, Computer Engineering · ORCID 0009-0004-8416-8368
ICRA 2026 Workshop on Reinforcement Learning for Imitation Learning (RL4IL)

Unitree G1 reaching toward a target during play evaluation (dual-critic policy, Isaac Lab).

Abstract

Multi-objective reinforcement learning for humanoid robots must coordinate locomotion and manipulation within a single policy. A natural design choice is whether to use a single (unified) critic that estimates the combined value of all objectives, or separate (dual) critics with disjoint reward signals. We present a controlled comparison on the Unitree G1 humanoid (23 active DoF) in NVIDIA Isaac Lab, training loco-manipulation policies through a sequential curriculum spanning 13 levels from stationary reaching to walking with variable-orientation targets. In standardized evaluation, dual-critic policies reach targets 3.5× faster (6.5 vs. 22.6 simulation steps), achieve 2× higher throughput (14.3 vs. 7.0 validated reaches per 1,000 steps), and attain higher validated reach rates (65.2% vs. 53.8%) compared to the unified-critic policy. Notably, additional anti-gaming reward mechanisms provide no further improvement beyond the architectural change alone (60.9% vs. 65.2%). These results have direct implications for the emerging paradigm of RL fine-tuning of imitation-learned policies: when refining a pre-trained manipulation policy with RL, a unified critic risks suppressing the learned behavior through competing locomotion gradients. These findings demonstrate that critic architecture is a primary—and often overlooked—design choice in multi-objective humanoid RL, with greater impact than reward engineering on reaching efficiency.

The 13-level curriculum described above is the one the dual-critic run followed. The unified-critic run used a different, 40-level curriculum — see Experimental caveats below.

Results

Metric Unified critic (S6u) Dual critic (S6s) Dual + anti-gaming (S7)
Steps to target 22.6 6.5 5.8
Throughput (validated reaches / 1,000 steps) 7.0 14.3 13.0
Validated reach rate 53.8% 65.2% 60.9%

Standing evaluation: 3,000 steps, one environment, deterministic actions, seed 42. Adding five anti-gaming reward mechanisms on top of the dual critic (S7) does not improve on the architectural change alone.

Experimental caveats

These numbers come from a single pair of training runs, and those runs differ in more than the critic. What follows is what was not held constant.

  • The two runs did not train on the same curriculum. The unified-critic run used a 40-level ladder (reaching → orientation → gripper → height/payload); the dual-critic run used a 13-level one (standing → walking → fixed orientation → arbitrary orientation out to an 80° cone). These are different schedules, not long and short versions of one. When its checkpoint was saved the unified policy was at level 10 of 40 — standing still, 0.05 m tolerance, fixed palm-down orientation — while the dual policy had completed 12 of 12: walking at up to 0.6 m/s with an arbitrary commanded palm direction at 0.04 m tolerance.
  • Different locomotion reward. One of the fourteen locomotion weights differs: forward velocity tracking, 3.0 in the unified script against 5.0 in the dual one. This was active for the entire length of both runs.
  • Different arm action dimensionality. The unified policy's arm actor emits 12 values (5 arm + 7 finger) against the dual policy's 5. The seven finger outputs were sampled and entered the policy update, but never reached the robot — finger control activates at curriculum level 20 and the run ended at 10 — and they are discarded at evaluation. The confound is in exploration and policy entropy rather than in the task performed.
  • Single seed. The training scripts set no random seed at all, so each arm is one run with no variance estimate. The evaluation harness does seed itself, so the two policies are compared on matched target sequences.
  • Parallel environments. 2048 for the dual-critic run and 4096 for the anti-gaming run, both recorded. The unified run's launch flags were never saved; 2048 is inferred from a reach-counter ceiling argument, not read from a config.

What this means. The evaluation itself is apples-to-apples — one harness, one shared locomotion branch, matched target sequences — so the measured gap between these two checkpoints is real. The causal claim is not. With the curricula laid side by side, the simpler explanation for the gap is training progress: the two policies finished on tasks of very different difficulty, and the slower one had last been trained to stand still.

Critic architecture is therefore a hypothesis here, not an established cause. A controlled ablation — one curriculum, one action space, one reward set, only the critic swapped, across several seeds — is underway; these runs should be read as its starting point. Every training script now writes a full run_config.json so that any future pair can be checked field by field before being called a comparison.

Citation

@article{yardimci2026critic,
  title   = {Critic Architecture Matters: Dual vs. Unified Critics for
             Humanoid Loco-Manipulation},
  author  = {Yard{\i}mc{\i}, Mehmet Turan},
  journal = {arXiv preprint arXiv:2606.11891},
  year    = {2026},
  doi     = {10.48550/arXiv.2606.11891},
  url     = {https://github.com/mturan33/isaac-g1-ulc},
  note    = {ICRA 2026 Workshop on Reinforcement Learning for
             Imitation Learning (RL4IL)}
}