Skip to the content.

Video-Touch

NEW! Check a blog post on Google Developers about our project!

Description

Video-Touch: Remote Robot Control by DNN-based Gesture Recognition

We present Video-Touch, a breakthrough technology for multi-user and real-time control of robot by DNN-based gesture recognition. The users can have a video conferencing in a digital world and at the same time to perform dexterous manipulations with tangible objects by remote robot. As the scenario, we proposed the remote robotic COVID-19 test Laboratory to substitute medical assistant working in protective gear in close proximity with infected cells and to considerably reduce the time to receive the test results. The proposed technology suggests a new type of reality, where multi-users can jointly interact with remote object (e.g. make a new building design, joint cooking in robotic kitchen, etc), and discuss/modify the results at the same time.

video_touch_scheme

System Overview

We were wondering if it is even possible to control a robot remotely using only your own hands - without any additional devices like gloves or a joystick - not suffering from a significant delay. We decided to use computer vision to recognize movements in real-time and instantly pass them to the robot. Thanks to MediaPipe now it is possible.

Our system looks as follows:

  1. Video conference application gets a webcam video on the user device and sends it to the robot computer (“server”);
  2. User webcam video stream is being captured on the robot's computer display via OBS virtual camera tool;
  3. The recognition module reads user movements and gestures with the help of MediaPipe and sends it to the next module via ZeroMQ;
  4. The robotic arm and its gripper are being controlled from Python, given the motion capture data.

robotic-arm

Live Demos

This project has had a great reception not only in ​robotics but also in other areas such as life sciences, art, and medicine. So much so that it has been featured in various conferences, festivals, and television channels.

1. SIGGRAPH Asia 2020

IMAGE_ALT

2. Russia 24 TV Channel

IMAGE_ALT

3. Russia 1 TV Channel

IMAGE_ALT

4. PANGARDENIA ARS ELECTRONICA 2020 Festival for Art, Technology & Society Saint-Petersburg

IMAGE_ALT

Authors

Citation

ZoomTouch: Multi-User Remote Robot Control in Zoom by DNN-based Gesture Recognition
Ilya Zakharkin, Arman Tsaturyan, Miguel Altamirano Cabrera, Jonathan Tirado and Dzmitry Tsetserukou
in SIGGRAPH Asia 2020 Emerging Technologies
arXiv ACM Procedings Siggraph Asia 2020

BibTeX

@article{zakharkin2020zoomtouch,
  title={ZoomTouch: Multi-User Remote Robot Control in Zoom by DNN-based Gesture Recognition},
  author={Zakharkin, Ilya and Tsaturyan, Arman and Altamirano-Cabrera, Miguel and Tirado, Jonathan and Tsetserukou, Dzmitry},
  booktitle = {SIGGRAPH Asia 2020 Emerging Technologies},
  year={2020}
}

Support or Contact

Skolkovo Institute of Science and Technology, Bolshoy Boulevard 30, bld. 1, Moscow, Russia 121205