smolvla_flow_rl
Online RL fine tuning of flow matching VLA policies on a single 12GB consumer GPU.
No results are reported here or in the repository. Every number published so far describes the machinery, what it trains, what it weighs and how long it takes, and none describes how well it works. Measurements made with this substrate will be released with the accompanying paper.
Who is this for?
If you are trying to fine tune SmolVLA with reinforcement learning, run PPO on a flow matching policy, or do VLA RL on limited hardware like a laptop GPU, this substrate is built for exactly that situation. It assumes you already have a pretrained policy and want it to improve from its own experience rather than from more demonstrations. It does not assume a cluster, and it does not assume you will take the training loop on trust: the failures that would otherwise pass silently are the ones it reports on.
Can a VLA policy be fine tuned with online RL on a single consumer GPU?
Yes, and this repository is the machinery for doing it. What makes VLA RL on a consumer GPU fit is memory discipline rather than a new algorithm: the simulator runs in separate processes, the update grades only the part of the action chunk the environment executed, and the critic is kept outside the pretrained action head. SmolVLA reinforcement learning fine tuning is the case it was built for, and everything here was developed and run on one 12GB laptop card.
How do you get action log probabilities from a flow matching policy?
A flow matching policy produces an action chunk by integrating a learned velocity field from noise. Integrating it deterministically gives no action likelihood, so a policy gradient has nothing to act on. This repository takes the standard route around that, following the approach of pi_RL: one step of the integration is treated as stochastic, so the sampled chunk has a tractable log probability. The pretrained action head stays bit identical and is driven through a callback, and what the update then performs is flow matching PPO, a clipped policy gradient over the action the environment actually executed.
What is inside the substrate?
Seven parts, each of which a run of this kind needs in order to be trustworthy rather than merely runnable.
- A stochastic sampler for flow matching policies that leaves the pretrained action head bit identical, driving it through a callback rather than modifying it.
- A policy gradient update that grades the action the environment actually executed, over the executed part of the chunk only, with the likelihood ratio and its spread recorded once per update cycle.
- A critic head with adaptive target rescaling, kept outside the action head.
- A parameter manifest and a learning rate proof printed before the first update, so what is training and at what rate is a logged fact rather than an assumption.
- A process isolated simulator harness with a sparse terminal reward and no shaping.
- A deterministic evaluation entry point that reads several horizons out of one run, with a completeness gate that refuses to produce a number from an unfinished evaluation.
- A verification protocol whose rules are written down, with the subset that can be checked mechanically wired into a single gate entry point.
The harness drives LIBERO, so LIBERO RL fine tuning is what it is set up for out of the box, and the pieces above are what a run reports about itself while doing it.
What hardware does it need?
It was developed and tested on a single RTX 5070 Ti laptop GPU with 12GB of VRAM, under WSL2. Anything with 12GB or more should behave the same, and less is untested. It needs Linux with a CUDA card and an EGL capable driver, since EGL is the simulator's headless path.
How does this relate to pi_RL, RLinf, LeRobot and SmolVLA?
The training formulation is the one pi_RL introduced for flow based vision language action models, so this is
the pi_RL SmolVLA case wired for one card. Four files are derivative works of RLinf under the Apache License
2.0, and two of those additionally follow LeRobot's own policy implementation, so they have two upstreams each.
SmolVLA is the policy family this drives and nothing here reimplements it, LIBERO is the benchmark the harness
drives, and no source file from any of them is redistributed here. What came from where is recorded in
NOTICE and CHANGES_FROM_UPSTREAM.md, and every adapted file carries a notice of
modification in its header.
How do I cite it?
Use the Cite this repository box in the right hand column of the
GitHub page. It is generated from
CITATION.cff and gives both APA and BibTeX. That file also carries the upstream work, so it points
a reader at RLinf, pi_RL, LeRobot, SmolVLA and LIBERO as well.
Does it work with diffusion policies?
It is built and tested for flow matching policies, and diffusion is not targeted. The likelihood this trains on comes from treating one step of a flow integration as stochastic; a diffusion policy has its own noise schedule and would need its own path to the same thing. Nothing here forbids that adaptation, but none of it has been tried, so treat it as unwritten rather than as supported.
Can it run on 8GB?
Untested. 12GB is the configuration everything was developed and run on, and anything at or above that should behave the same. Below it nobody has tried, so the honest answer is that it might fit with a shorter rollout and a smaller batch, or it might not fit at all, and this page is not the place to guess.