A convolutional neural network with feature fusion for real-time hand posture recognition

Gesture based human–computer interaction is both intuitive and versatile, with diverse applications such as in smart houses, operating theaters and vehicle infotainment systems. This paper presents a novel architecture, combining a convolutional neural network (CNN) and traditional feature extractors, capable of accurate and real-time hand posture recognition. The proposed architecture is evaluated on three distinct benchmark datasets and compared with the state-of-the art convolutional neural networks. Extensive experimentation is conducted using binary, grayscale and depth data, as well as two different validation techniques. The proposed feature fusion-based convolutional neural network (FFCNN) is shown to perform better across combinations of validation techniques and image representation. The recognition rate of FFCNN on binary images is equivalent to grayscale and depth when the aspect ratio of gestures is preserved. A real-time recognition system is presented with a demonstration video.