Critic Architecture Matters:
Dual vs. Unified Critics for Humanoid Loco-Manipulation

Mehmet Turan Yardımcı
Çukurova University, Computer Engineering · ORCID 0009-0004-8416-8368
ICRA 2026 Workshop on Reinforcement Learning in the Era of Imitation Learning (RL4IL)

Unitree G1 reaching toward a target during play evaluation (dual-critic policy, Isaac Lab).

Abstract

Multi-objective reinforcement learning for humanoid robots must coordinate locomotion and manipulation within one policy. A natural design choice is between a single (unified) critic that estimates the combined value of all objectives and separate (dual) critics with disjoint reward signals. We compare the two on the Unitree G1 humanoid in NVIDIA Isaac Lab. In the standing mode of a standardized evaluation, the dual-critic run reaches targets 3.5× faster (6.5 vs. 22.6 simulation steps), achieves 2× the throughput (14.3 vs. 7.0 validated reaches per 1,000 steps) and a higher validated reach rate (65.2% vs. 53.8%) than the unified-critic run. That evaluation pins the fingers open for every policy, whereas the unified run had trained driving its own. When the unified run drives its own fingers, with nothing else changed, the standing-mode gap falls from 3.5× to 1.3× in speed and from 2× to 1.1× in throughput. This is a single re-evaluation of a single checkpoint, and we do not generalize from it. Adding five anti-gaming reward mechanisms to the dual critic did not raise validated reach rate (60.9% vs. 65.2%). The two runs differ not only in the critic but also in the PPO update rule (one summed advantage under one likelihood ratio, versus a per-stream advantage and a ratio per actor), and further in curriculum, arm action dimensionality, finger control and reward weights; each is a single run. The measurement therefore cannot separate the critic from the update rule. We argue that critic architecture deserves explicit treatment as a design variable in multi-objective humanoid RL, and specify the single-variable ablation needed to establish its causal contribution.

Results

Standing evaluation Unified critic (S6u) S6u, own fingers driven‡ Dual critic (S6s) Dual + anti-gaming (S7)
Steps to target 22.6 8.3 6.5 5.8
Throughput (validated reaches / 1,000 steps) 7.0 12.7 14.3 13.0
Validated reach rate 53.8% 59.4% 65.2% 60.9%

Standing evaluation: 3,000 steps, one environment, deterministic actions, seed 42; the original benchmark pins the fingers open for every policy. ‡Same unified checkpoint, same benchmark, seed and step count; the only change is that its seven finger outputs drive the fingers, as they did in training. This is a single re-evaluation of a single checkpoint, and we do not generalize from it; it still runs on the dual run's locomotion branch, not its own, and that second mismatch is unmeasured. The anti-gaming variant (S7) is faster to target than S6s but did not raise validated reach rate or throughput. Walking-mode numbers and the rest of the table are in the paper (Table II).

Experimental caveats

These numbers come from a single pair of training runs, and those runs differ in more than the critic. What follows is what was not held constant.

  • The two runs did not train on the same curriculum. The unified-critic run used a 40-level ladder (reaching → orientation and gripper together → commanded torso height → payload); the dual-critic run used a 13-level one (standing → walking → fixed orientation → arbitrary orientation out to an 80° cone). These are different schedules, not long and short versions of one. When its checkpoint was saved the unified policy was at level 10 of 40 — standing still, 0.05 m tolerance, fixed palm-down orientation at 0.8 rad slack, with gripper control active — while the dual policy had completed 12 of 13: walking at up to 0.6 m/s with an arbitrary commanded palm direction at 0.04 m tolerance.
  • Different reward tables. The two scripts do not share a reward table. Forward velocity tracking is weighted 2.5 in the unified script against 5.0 in the dual one, and the term lists differ: the dual configuration adds leg posture, standing still and foot stability, the unified one gripper, height tracking and payload terms. Active for the entire length of both runs.
  • Different update rule. The difference is not only the critic. The unified script normalizes one summed advantage and applies a single likelihood ratio to both actors; the dual script forms a separate advantage per reward stream and updates each actor with its own ratio, clipping and optimizer. Critic architecture and credit assignment move together here, so the measured gap cannot separate them.
  • Different arm action dimensionality, and a train/eval mismatch. The unified policy's arm actor emits 12 values (5 arm + 7 finger) against the dual policy's 5. Finger control activates at curriculum level 10, so those seven outputs drove the robot's fingers for 18,838 of the 20,000 training iterations. They are discarded in the original, pinned-finger evaluation, where the benchmark pins the fingers open for every policy, so the unified policy is measured in a configuration it did not train in. The confound is in the task performed and may also lie in exploration and policy entropy, which were not measured. When the unified checkpoint drives its own fingers, with nothing else changed, the standing-mode gap falls from 3.5× to 1.3× in speed and from 2× to 1.1× in throughput (walking: 2.3× to 1.2× and 1.8× to 1.03×). One re-evaluation of one checkpoint; not generalized.
  • Single run, no seed. Neither training run set a random seed, so each arm is one run with no variance estimate. The evaluation harness does seed itself, so the two policies are compared on matched target sequences.
  • Locomotion branch at evaluation. The harness gives all three policies the locomotion branch the dual runs share; the unified run's own locomotion weights differ from it, so its arm is evaluated with a locomotion branch it did not train with. What that costs has not been measured, and the finger re-evaluation leaves it in place.
  • Parallel environments. 2048 for the dual-critic run and 4096 for the anti-gaming run, both recorded. The unified run's launch flags were never saved; 2048 is inferred from a reach-counter ceiling argument, not read from a config.

What this means. The evaluation is internally consistent — one harness, one shared locomotion branch, matched target sequences — so the measured gap is a real property of these two checkpoints under this harness. It depends on the harness: with the unified run's finger outputs restored and nothing else changed, the standing-mode gap falls from 3.5× to 1.3× in speed and from 2× to 1.1× in throughput. For the difference that remains, training progress is at least as parsimonious an explanation as critic architecture, and the unmeasured locomotion-branch mismatch is a third candidate; this design cannot isolate the critic's contribution from the other differences listed above.

Critic architecture is therefore a hypothesis here, not an established cause. A controlled ablation — one curriculum, one action space, one reward set, one update rule, only the critic swapped, across several seeds — is specified and instrumented but not yet run; these runs should be read as its starting point. Every training script now writes a full run_config.json so that any future pair can be checked field by field before being called a comparison.

Citation

@article{yardimci2026critic,
  title   = {Critic Architecture Matters: Dual vs. Unified Critics for
             Humanoid Loco-Manipulation},
  author  = {Yard{\i}mc{\i}, Mehmet Turan},
  journal = {arXiv preprint arXiv:2606.11891},
  year    = {2026},
  doi     = {10.48550/arXiv.2606.11891},
  url     = {https://github.com/mturan33/isaac-g1-ulc},
  note    = {ICRA 2026 Workshop on Reinforcement Learning in the Era
             of Imitation Learning (RL4IL)}
}