In this paper, we propose a vision-based hand gesture recognition system for human-computer interaction. The gesture recognition systems are employed in developing a rock-paper-scissors game between human and our robotic hands in realtime. Our task is to predict the gestures as soon as possible by using high-speed cameras. Due to the computational complexity, the standard long-term recurrent convolution network-based action classification system cannot be contented with classification tasks based on high-speed cameras. We propose to address this issue by employing a more efficient network architecture and using a threshold-based method to predict the gesture in advance. We validate our proposed method on the new gesture dataset for the rock-paper-scissors game. The model is able to successfully learn gestures varying in duration and complexity. A comparative analysis of CNN and long-term recurrent convolution network is performed. We report a gesture classification accuracy of 97% and report a near real-time computational complexity of 7 ms per frame.