rps-cv

A Rock-Paper-Scissors game using computer vision and machine learning on Raspberry Pi.

By Julien de la Bruère-Terreault (drgfreeman@tuta.io)

Animated screenshot
Click on image to access video on YouTube.

This project is showcased in issue 74 of the MagPi, the official magazine of the Raspberry Pi Foundation.

Summary

Project origin

This project results from a challenge my son gave me when I was teaching him the basics of computer programming making a simple text based Rock-Paper-Scissors game in Python. At that time I was starting to experiment with computer vision with a Raspberry Pi and an old USB webcam so my son naively asked me:

"Could you make a Rock-Paper-Scissors game that uses the camera to detect hand gestures?"

I accepted the challenge and about a year and a lot of learning later, I completed the challenge with a functional game.

Overview of the game

The game uses a Raspberry Pi computer and Raspberry Pi camera installed on a 3D printed support with LED strips to achieve consistent images.

The pictures taken by the camera are processed and fed to an image classifier that determines whether the gesture corresponds to "Rock", "Paper" or "Scissors" gestures.

The image classifier uses a Support Vector Machine, a class of machine learning algorithm. The image classifier has been priorly "trained" with a bank of labeled images corresponding to the "Rock", "Paper", "Scissors" gestures captured with the Raspberry Pi camera.

How it works

The image below shows the processing pipeline for the training of the image classifier (top portion) and the prediction of gesture for new images captured by the camera during the game (bottom portion). Click here for full size image. Rock-Paper-Scissors computer vision & machine learning pipeline

Dependencies

The project depends on and has been tested with the following libraries:

* Follow this guide for installation of OpenCV on the Raspberry Pi. Install Python libraries within the same virtual environment as OpenCV using the pip install <package_name> command. Picamera is installed by default on Raspbian images.

Hardware:

Camera & lighting setup Camera & lighting setup Camera & lighting setup

Program files

* Note that the due to memory limitations on the Raspberry Pi, the train.py script may not run properly on the Raspberry Pi with training sets of more than a few hundred images. Consequently, it is recommended to run these on a more powerful computer. This computer must also have OpenCV, Python 3.4+ and the numpy, scikit-learn and scikit-image Python libraries installed.

Library modules

Ouput & Screenshots

Training mode

Typical output from train.py (on PC with Intel Core I7-6700 @3.4GHz, 16GB RAM, Anaconda distribution):

(rps-cv) jul@rosalind:~/pi/git/rps-cv$ python train.py
+0.0: Importing libraries
+3.75: Generating image data
Completed processing 1708 images
  rock: 562 images
  paper: 568 images
  scissors: 578 images
+99.51: Generating test set
+99.64: Defining pipeline
+99.64: Defining cross-validation
+99.64: Defining grid search
Grid search parameters:
GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=42, shuffle=True),
       error_score='raise',
       estimator=Pipeline(steps=[('pca', PCA(copy=True, iterated_power='auto', n_components=None,
       random_state=None, svd_solver='auto', tol=0.0, whiten=False)), ('clf', SVC(C=1.0,
       cache_size=200, class_weight=None, coef0=0.0, decision_function_shape=None, degree=3,
       gamma='auto', kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True,
       tol=0.001, verbose=False))]),
       fit_params={}, iid=True, n_jobs=4,
       param_grid={'clf__C': array([   1.     ,    3.16228,   10.     ,   31.62278,  100.     ]),
                   'clf__gamma': array([ 0.0001 ,  0.00032,  0.001  ,  0.00316,  0.01   ]),
                   'pca__n_components': [60]},
       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,
       scoring='f1_micro', verbose=1)
+99.64: Fitting classifier
Fitting 5 folds for each of 25 candidates, totalling 125 fits
[Parallel(n_jobs=4)]: Done  42 tasks      | elapsed:  2.1min
[Parallel(n_jobs=4)]: Done 125 out of 125 | elapsed:  5.9min finished
Grid search best score: 0.9910406616126809
Grid search best parameters:
  pca__n_components: 60
  clf__C: 10.0
  clf__gamma: 0.00031622776601683794
+458.66: Validating classifier on test set
Classifier f1-score on test set: 0.9922178988326849
Confusion matrix:
[[84  1  0]
 [ 1 84  0]
 [ 0  0 87]]
Classification report:
             precision    recall  f1-score   support

       rock       0.99      0.99      0.99        85
      paper       0.99      0.99      0.99        85
   scissors       1.00      1.00      1.00        87

avg / total       0.99      0.99      0.99       257

+458.72: Writing classifier to clf.pkl
+467.25: Done!

Play mode with Graphical User Interface (playgui.py)

Initial screen:
Initial screen

Computer wins the play:
Computer wins play

Player wins the play:
Player wins play

Tie:
Tie

Game over, player wins the game:
Game over - player wins

Image capture mode (capture.py)

Capture mode

Play mode without GUI (play.py)

Play no GUI