Interactive Rock Paper Scissors using TensorFlow.js

TensorFlow recently released TensorFlow.js – a WebGL accelerated, browser based JavaScript library for training and deploying ML models. This library can be used to define, train, and run machine learning models entirely in the browser, using Javascript and a high-level layers API. Some of the advantages of using an in browser machine learning framework include the fact that there’s no need to install any libraries or drivers so that application can be lightweight. Users can also open a web site on a mobile device. All data stays on the client side, useful for building secure applications. Some other applications that can be built using this framework include interactive educational applications and games.

TensorFlow.js has provided several examples – you can also see live demos here. I have modified the pacman example to build a Rock-Paper-Scissors game. Here’s a quick recording of the game.

Here’s whats happening under the covers:

    Loading a mobilenet model that’s pretrained – these models are based on convolutional neural network designed by researches at Google. They are “mobile-first” so are resource-friendly and execute fast. They are fast, small and pretty accurate.
    Now we build an image classifier that can detect if an image is rock, paper or scissors. Since we’re tackling a custom problem, we need to start with creating our dataset – this is where the dozen or so images of rock, paper and scissors were taken using the webcam and labeled appropriately.
    Final step is to retrain the mobilenet model – luckily for us TensorFlow comes packaged with great tools that you can use to retrain MobileNets without having to actually write any code.

We will be hosting this application on AppEngine shortly.

PS: As an aside, here’s a rock-paper-scissors app built by my daughter on scratch – https://scratch.mit.edu/projects/213853734/ – check it out 🙂