|
|
|
|
|
|
|
|
|
|
|
|
| Overview of the NL2Contact setting. Our method learns to model 3D hand-object contact from language description. The modeling contact can be used to grasp pose optimization. By utilizing the predicted contact to refine the initial grasp, our performance surpasses that of ContactOpt and ContactPose. It's worth noting that ContactPose is the annotation from the motion capture system. |
| Modeling the physical contacts between the hand and object is standard for refining inaccurate hand poses and generating novel human grasp in 3D hand-object reconstruction. However, existing methods rely on geometric constraints that cannot be specified or controlled. This paper introduces a novel task of controllable 3D hand-object contact modeling with natural language descriptions. Challenges include i) the complexity of cross-modal modeling from language to contact, and ii) a lack of descriptive text for contact patterns. To address these issues, we propose NL2Contact, a model that generates controllable contacts by leveraging staged diffusion models. Provided with a language description of the hand and contact, NL2Contact generates realistic and faithful 3D hand-object contacts. To train the model, we build ContactDescribe, the first dataset with hand-centered contact descriptions. It contains multi-level and diverse descriptions generated by large language models, based on carefully designed prompts (e.g. grasp action, grasp type, contact location, free finger status). We show applications of our model to grasp pose optimization and novel human grasp generation, both based on a textual contact description. |
 |
| NL2Contact pipeline. We propose a novel method to model 3D hand-object contact using language description. Our framework is composed of 1) a Text-to-Hand-Object Fusion module, efficiently fusing the geometry information from both point clouds and semantic information from natural language, 2) a staged latent diffusion model to first generate the hand pose fitting the hand description and then generate the 3D hand-object contact map conditioned on the descriptions and the generated hand pose. 3) contact optimization, which iteratively optimizes the hand pose using the generated contact. We also added a switch to toggle between its application for grasp pose optimization and human grasp generation. |
 |
| An example of the dataset with multi-level language descriptions. Given the proposed hand-centered contact prompts, our dataset leverages ChatGPT3 to generate multi-level and diverse free-form text descriptions of contact patterns. |
|
|
|
Acknowledgements |
