A sample project implementing TensorFlowSharp and a trained Convolutional Neural Network (CNN) to classify MNIST images in the Unity Video Game Engine. TensorFlow models can be applied to Unity for recognition games and simulations. This example takes in sample image textures, evaluates them using a trained model, and displays the classification to the screen.
The following packages are required for this project. Specific TensorFlow and Keras versions are required. If you'd like to keep a newer version of these packages, install these packages in a virtual environment. If you don't wish to use a virtual environment, skip the following section.
To create a new virtual environment for TensorFlow 1.4. Note: your python3 library may differ.
cd ~/
virtualenv -p /usr/bin/python3 tensorflow_1.4To activate the virtual environment and use for training.
source ~/tensorflow_1.4/bin/activateTo deactivate this virtual environment, once finished with training.
deactiveWarning: running the following script will install TensorFlow, Keras, and python packages. No su permissions are necessary.
chmod +x TensorFlow/install.sh
TensorFlow/install.shHow to install all of the packages manually (without install script).
Install version 1.4 of TensorFlow (cpu or gpu). Version 1.4 is required for the TFSharp Unity package.
pip3 install tensorflow==1.4Install version 2.1.3 of Keras. This version is one compatible with TensorFlow 1.4.
pip3 install keras==2.1.3Install argparse, matplotlib, and pillow.
pip3 install argparse
pip3 install matplotlib
pip3 install pillowThe Unity project is located in the following folder: Unity-Files/MNIST. Open this folder from Unity to get started. The example scene can be found in Scenes > Classifier.
Run the following python script to train the CNN model.
cd TensorFlow/
python3 mnist_cnn1.pyParameters that can be run with this script:
--model_name # the name of the saved model default='mnist_cnn1'
--export_images # instead of training a model, save MNIST to .png's default=False
--export_number # number of MNIST images to export (if enabled) default=10
--plot_images # instead of training a model, display MNIST images default=False
--epochs # number of epochs to train model for default=5
--batch_size # batch size to use for training default=128These parameters can be applied to the python script in the following way:
python3 mnist_cnn1.py --[param1_name]=[param1_value] --[param2_name]=[param2_value]The python script will generate a graph of the TensorFlow model once training is complete. Two files of importance are saved: frozen_mnist_cnn1.bytes and opt_mnist_cnn1.bytes. The latter is an optimized (smaller) version of the former. The model files are saved to the following location:
TensorFlow/out/frozen_mnist_cnn1.bytes
TensorFlow/out/opt_mnist_cnn1.bytesTo add these files to Unity, run the following while inside the parent directory of the project.
cp TensorFlow/out/*.bytes Unity-Files/MNIST/Assets/TensorFlow/Frozen_graphs/Add either of the .bytes files to the Graph Model element in the MNIST_Classifier object. Add any image textures you wish to test with the model to the Input Textures element. See the following for a working example setup.
In order to process the Unity TensorFlow Plugin, you must perform the following under Edit > Project Settings > Player > Other Settings
Press play, and use the arrow keys to change the input image to the TensorFlow model.
Please read CONTRIBUTING.md for details on the code of conduct, and the process for submitting pull requests.
This project is licensed under the Apache 2.0 License - see the LICENSE file for details.