It is important for a robot not only to learn to perform a task but also not to go beyond reasonable limits. Until now, engineers had to choose: either reinforcement learning (a method in which a robot masters behavior through trial and error, like in the game "hot and cold"), or stability guarantees. Researchers from the Grainger College of Engineering at the University of Illinois Urbana-Champaign combined both approaches.
The work "Contraction-Aware Reinforcement Learning for Nonlinear Control with Statistical Robustness" was published in IEEE Transactions on Robotics. The authors are Minjae Cho, Hiroyasu Tsukamoto and Huy T. Tran.
"Reinforcement learning gives robots a promising way to master efficient behavior, but it often lacks stability guarantees," says graduate student Minjae Cho. "Without confidence that the robot will remain stable, deploying it in practice is difficult. In the worst case, it could break down, and repairs would be expensive."
The new method is called CARL (Contraction-Aware Reinforcement Learning). It combines reinforcement learning with contraction theory. This mathematical theory makes it possible to set limits within which tracking errors can vary: small deviations are possible, but the method guarantees that they will not exceed certain bounds. Thanks to this, reinforcement learning can act as the robot's primary controller without losing stability.
Cho began with simulations and then moved to the aerospace engineering laboratory, where he tested CARL on real robots. "A model in a computer and a physical robot differ. For example, in the simulator the mass is 1 kg, but in reality it is 1.2 kg. This is called the sim-to-real gap, but our algorithm performed better under such uncertainties," the researcher explains.
The experiments were conducted with three types of robots: the wheeled TurboBot 3 Burger, a flapping flying drone and the six-legged Jethexa. A video overlaying the trajectories of the old algorithms and CARL shows that the new method keeps the robots on course more stably and brings them to the goal faster.
"There was no guarantee that our method would work better than existing ones, especially given the increasing complexity of robots. But it turned out that integrating the two approaches is not only novel but also consistently delivers a better result - up to 15% - both in simulation and in real-world tests," Cho notes.
One possible application is fighting forest fires: robots can be sent into dangerous zones to search for ignition points or deliver cargo. The CARL code is open and available on GitHub.