Abstract
In the light of factories of the future, we present a reliable framework for real-time safe physical human-robot collaboration using static hand gestures. To ensure productive and safe interaction between robot and human coworkers, it is imperative that the robot extracts the essential information about the human coworker. We address this by designing a framework for safe and intuitive robot programming based on hand gesture recognition. First, the OpenPose library is integrated with Microsoft Kinect V2, to obtain a 3D estimation of the human skeleton. With the help of 10 volunteers, we record an image dataset of alphanumeric static hand gestures, taken from the American Sign Language. We name our dataset as OpenSign and release it to the community for bench-marking. The Inception-v3 convolutional neural network is adapted to train the hand gesture detector. To augment the data for training a hand gesture detector, we use OpenPose to localize the hands in the dataset images and segment the backgrounds of hand images using the Kinect depth map. Then, the backgrounds are substituted with random patterns and indoor architecture templates. Fine-tuning of Inception V3 is performed in three phases, to achieve validation accuracy of 99.1% and test accuracy of 98.9%. An asynchronous integration of image acquisition and hand gesture detection is performed to ensure real-time detection of hand gestures. Finally, the proposed framework is integrated in our physical human-robot interaction library OpenPHRI. Using OpenPHRI, we validate the performance of the proposed framework through a complete teaching by demonstration experiment with a robotic manipulator.