Video-Touch
NEW! Check a blog post on Google Developers about our project!
Description
Video-Touch: Remote Robot Control by DNN-based Gesture Recognition
We present Video-Touch, a breakthrough technology for multi-user and real-time control of robot by DNN-based gesture recognition. The users can have a video conferencing in a digital world and at the same time to perform dexterous manipulations with tangible objects by remote robot. As the scenario, we proposed the remote robotic COVID-19 test Laboratory to substitute medical assistant working in protective gear in close proximity with infected cells and to considerably reduce the time to receive the test results. The proposed technology suggests a new type of reality, where multi-users can jointly interact with remote object (e.g. make a new building design, joint cooking in robotic kitchen, etc), and discuss/modify the results at the same time.
System Overview
We were wondering if it is even possible to control a robot remotely using only your own hands - without any additional devices like gloves or a joystick - not suffering from a significant delay. We decided to use computer vision to recognize movements in real-time and instantly pass them to the robot. Thanks to MediaPipe now it is possible.
Our system looks as follows:
- Video conference application gets a webcam video on the user device and sends it to the robot computer (“server”);
- User webcam video stream is being captured on the robot's computer display via OBS virtual camera tool;
- The recognition module reads user movements and gestures with the help of MediaPipe and sends it to the next module via ZeroMQ;
- The robotic arm and its gripper are being controlled from Python, given the motion capture data.
Live Demos
This project has had a great reception not only in robotics but also in other areas such as life sciences, art, and medicine. So much so that it has been featured in various conferences, festivals, and television channels.
1. SIGGRAPH Asia 2020
2. Russia 24 TV Channel
3. Russia 1 TV Channel
4. PANGARDENIA ARS ELECTRONICA 2020 Festival for Art, Technology & Society Saint-Petersburg
Authors
- Ilya Zakharkin, Skolkovo Institute of Science and Technology, Moscow Institute of Physics and Technology (State University), Russia
- Arman Tsaturyan, Skolkovo Institute of Science and Technology, Moscow Institute of Physics and Technology (State University), Russia
- Miguel Altamirano Cabrera, Skolkovo Institute of Science and Technology, Russia
- Jonathan Tirado, Skolkovo Institute of Science and Technology, Russia
- Dzmitry Tsetserukou, Skolkovo Institute of Science and Technology, Russia
Citation
ZoomTouch: Multi-User Remote Robot Control in Zoom by DNN-based Gesture Recognition
Ilya Zakharkin, Arman Tsaturyan, Miguel Altamirano Cabrera, Jonathan Tirado and Dzmitry Tsetserukou
in SIGGRAPH Asia 2020 Emerging Technologies
arXiv
ACM Procedings
Siggraph Asia 2020
BibTeX
@article{zakharkin2020zoomtouch,
title={ZoomTouch: Multi-User Remote Robot Control in Zoom by DNN-based Gesture Recognition},
author={Zakharkin, Ilya and Tsaturyan, Arman and Altamirano-Cabrera, Miguel and Tirado, Jonathan and Tsetserukou, Dzmitry},
booktitle = {SIGGRAPH Asia 2020 Emerging Technologies},
year={2020}
}
Support or Contact
Skolkovo Institute of Science and Technology, Bolshoy Boulevard 30, bld. 1, Moscow, Russia 121205