Astribot releases SmoothRL, an asynchronous online RL framework for robots that can't wait for models

Astribot's foundation-model team has released SmoothRL, an online reinforcement learning framework designed to run asynchronously, with a technical report now live on the company's research page (astribot.com/en/research/SmoothRL). The problem it targets is concrete: real robots have no pause button. As VLA and world-action models generate longer action chunks and inference grows slower, a robot cannot stop every few hundred milliseconds to wait for the model — in highly dynamic tasks like throwing, a single pause can destroy accumulated speed and fail the whole attempt.

In real deployments the common pattern is the opposite: the robot keeps executing the current motion while the model computes the next chunk in the background. But asynchronous execution breaks the tidy assumption behind conventional online RL that what the model plans is what the robot does: within one action chunk, some actions can no longer be changed, some are truly executed, and some never happen at all. SmoothRL first splits the chunk by execution state into Committed, Execution and Discarded regions, and lets value gradients flow only through the Execution region — the actions the robot actually performed.

The second principle is that training and deployment must obey the same timing. SmoothRL runs asynchronous inference directly during training rollouts, so the replay buffer records trajectories produced under the real time relationship between computation and execution. Concretely, the team builds on task-specific fine-tuned base policies with RLT's skeleton, using a lightweight TD3-style actor-critic to predict residual corrections in raw action space. On the Astribot S1, the robot executes at 30 Hz while inference is requested at 5 Hz, or one new action chunk every 200 ms; the base policy predicts 32 frames per chunk, of which Committed plus Execution cover 12 frames — about 6 truly executed — while the remaining 20 are overwritten by the next chunk before execution.

The team validated SmoothRL on the cable-driven S1 across two complementary classes of real deployment difficulty. In dynamic throwing, success climbed from the base policy's 39% to 94% at the 250-rollout evaluation point; in pen capping, which tolerates only about 5 mm of relative pose error, success rose from 8% to 83%; and in box opening — inserting a roughly 1 mm blade into a 2-3 mm lid seam — the learning curve was not monotonic, dipping from 30% to 20% around 150 episodes before recovering to 90%.

More telling than the success rates is how the robot's failure modes changed. Systematic biases — failing to modulate release speed by target distance when throwing, a persistent leftward drift of the blade, overly similar motions for different pen-cap positions — were corrected one by one using real execution results. With a smoothness constraint added to the policy, one real autonomous throwing rollout showed a 52% drop in acceleration RMS and a 47% drop in jerk on the right end effector, meaning the motion was not only more accurate but also smoother and more continuous.

The framework is positioned as online post-training for deployment rather than teaching skills from scratch. Once robots truly enter the real world, the new problems are usually not about being unable to do a task at all, but being off by a few millimeters or half a beat, or becoming unstable when the object moves. The report is candid about limits: it currently uses sparse success/failure rewards and still allows operator intervention when needed, so this is not yet fully autonomous self-improvement; every chunk-level inference must finish within a preset latency budget, and the lightweight residual policy's expressiveness is bounded by the frozen base policy.

SmoothRL was led by Astribot's Wang Jianan, with Gao Guang and Nong Yuxuan as co-first contributors and Huang Baifu participating. The work continues the reasoning thread of Astribot's Lumo family of foundation models — from Lumo-1's explicit reasoning to Lumo-2's prediction of how actions change the world — and connects to the company's full-stack route in which agents handle interaction and task understanding, base models build general manipulation ability, and RL keeps correcting policies from real execution.

The team's next steps include larger policy updates, broader task distributions, and combining asynchronous execution with end-to-end optimization of generative policies. For physical AI, SmoothRL points to a deployment paradigm that is becoming increasingly concrete: once a robot starts working, training is not over.