Flow Multi-Support:
Flow Matching Imitation Learning
for Multi-Support Manipulation

Quentin Rouxel             Andrea Ferrari             Serena Ivaldi             Jean-Baptiste Mouret             Inria, CNRS, Université de Lorraine
Paper ArXiv HAL Video

Abstract


Humanoid robots could benefit from using their upper bodies for support contacts, enhancing their workspace, stability, and ability to perform contact-rich and pushing tasks. In this paper, we propose a unified approach that combines an optimization-based multi-contact whole-body controller with Flow Matching, a recently introduced method capable of generating multi-modal trajectory distributions for imitation learning. In simulation, we show that Flow Matching is more appropriate for robotics than Diffusion and traditional behavior cloning. On a real full-size humanoid robot (Talos), we demonstrate that our approach can learn a whole-body non-prehensile box-pushing task and that the robot can close dishwasher drawers by adding contacts with its free hand when needed for balance. We also introduce a shared autonomy mode for assisted teleoperation, providing automatic contact placement for tasks not covered in the demonstrations.

Multi-Support Experiments on Talos Humanoid Robot

Non-Prehensile Box Pushing (Autonomous)

Closing of Dishwasher Drawers (Autonomous)

When trained on the red T-box, the autonomous policy performs poorly out-of-distribution with the blue U-box




This is mitigated with Shared Autonomy / Assisted Teleoperation approach

The human operator commands the left hand, and the policy commands the right hand



Assisted Teleoperation Pushing the Blue U-Box

Assisted Teleoperation Closing Dishwasher Drawers

Methodology

Multi-Support Manipulation Tasks:

  • Use additional contact to extend manipulation capabilities
  • Reach further, be more stable, apply higher pushing forces
  • Leverage whole-body motion and multi-contact strategies

Main Ideas

  • Imitation learning from human demonstrations (behavior cloning)
  • Use human "common sense" to learn contact sequence and location
  • Solve out-of-distribution tasks not covered by demonstrations using shared autonomy assisted teleoperation

Imitation Learning with Flow Matching

  • The policy outputs a full command trajectory instead of a single action
  • Use Flow Matching, a generative method able to capture high-dimensional and multi-modal distributions
  • Simpler framework, slightly more performant than Diffusion for robotics grounded in optimal transport theory

Multi-Contact Low-Level Controller

  • Use our previous work SEIKO Retargeting and Controller
    https://hucebot.github.io/seiko_controller_website/
  • Retargeting from Cartesian to whole-body commands with feasibility constraints
  • Implement smooth contact switch
  • Regulate contact forces on our position-controlled robot

Comparison Between Flow Matching and Diffusion Methods

Simulated T-Box Pushing Task

Simulated Bimodal Contact Reaching Task

BibTeX

@article{rouxel2024flowmultisupport,
  title={Flow Matching Imitation Learning for Multi-Support Manipulation},
  author={Rouxel, Quentin and Ferrari, Andrea and Ivaldi, Serena and Mouret, Jean-Baptiste},
  year={2024},
}