Cost about $3,600, it can be used to record the movements of human fingers to train the robot for flexible operations.
And not remotely operated, it has a pair of special gloves that capture precise data on hand movements through various sensors. Compared with traditional vision-based motion capture technology, these gloves will not fail due to vision obstruction and are more suitable for use in daily activities.
Once data on hand movements is collected, the DexCap system uses a mini PC carried in a backpack to reconstruct a 3D scene from an RGB-D camera and align the motion data with it. In this way, a very accurate hand movement model can be obtained, which can be used for further robot training.
The main application scenarios of DexCap are still in robot learning. Using the data collected by DexCap, researchers can train robots to perform a variety of complex two-handed tasks, such as collecting tennis balls and packaging objects. The learning of these tasks is entirely based on data from human movements and does not require any remote operation data. In addition, DexCap also supports expanding the scope of data collection in the wild, allowing the skills learned by the robot to be generalized to unseen objects by interacting with multiple objects in diverse environments.
Currently DexCap cannot handle operations that require the application of force. To overcome this limitation, the team is exploring the introduction of human-in-loop correction into the DexCap system. This approach allows rapid corrections in practical applications, improving the robot’s ability to perform tasks that require fine force control, such as making tea or using scissors.
Can we use wearables to collect robot data without actual robots?
Data from: @chenwang_j
Yes! Wearing a pair of gloves!🧤
We are proud to launch DexCap, a portable hand motion capture system that collects 3D data (point cloud + finger movement) for training the robot’s dexterous hands. Everything is open source
Project address:https://dex-cap.github.io
Video: