I am glad to introduce you to the machine learning-related technology that I think is very interesting and cutting-edge, and that is TensorFlow.js. If you are interested in machine learning, or you are a web developer who often uses the JavaScript programming language, then it will be very suitable for you. I will briefly introduce it to you in the following chapters. I hope it can be helpful and enlightening to you.

What is TensorFlow.js?

TensorFlow.js is a machine learning library that has only been released and updated since 2018. It allows us to develop ML models in JavaScript, and use ML directly in the browser or in Node.js. It uses flexible and intuitive APIs to build models from scratch using the low-level JavaScript linear algebra library or the high- level layers API. In short, it is designed for browser-side machine learning.

Why we need it?

In my opinion, its biggest advantage is the ability to perform machine learning on the browser side, which means that it has all the advantages of the browser, such as the ease of deployment of the browser and the huge advantages of cross-platform. Although our different mobile devices have different operating systems, they can all run browsers. That is to say, as long as we have a browser, we can open the same page and perform the same operations.

With the advancement of mobile device hardware and the development of network technology, mobile computing has become more and more mainstream, and our mobile devices are also taking on more and more computing tasks. For machine learning applications, the cost of development and deployment is very high, especially for devices with different platforms and different operating systems. In addition, for some infrequently used machine learning applications, such as only using models for speculation, in this case, the complex configuration will greatly affect the enthusiasm of the application. Although there are some machine learning applications on the web, they only play a role in data collection, and the real model inference is still performed on the cloud server. When we consider the communication overhead and data security issues, it becomes important to perform machine learning inference or training locally.

Therefore, TensorFlow.js came into being. I am optimistic that there will be many machine learning applications based on it in the near future.

What can we do with TensorFlow.js?

• We can import an existing pre-trained model and use it for inference. If we have a previously trained model, such as a Keras model or a TensorFlow model, then we can convert it to the TensorFlow.js format and run it on the browser for inference.
• We can also train the model. We can import existing models to continue training. In some scenarios, we have a ready-made pre-trained model, we can load it and use local data for training to improve accuracy.
• We can also create new machine learning models, use APIs similar to other mature machine learning libraries, and directly define training and running models on the browser side.

A demo

The demo is available here. This is a famous game of Pacman, but the difference is that you can control the direction of Pacman in real time through the browser-side machine learning technology. It includes two steps, namely training and inference.

Model Training

First, you need to use the camera of the device to capture the gesture pictures representing the four directions. The larger frame on the left is your current gesture, and the right corresponds to the four directions. When you collect pictures in a certain direction, first select the direction, then pose, and click the viewfinder in that direction to capture the picture. This is the training data. When you have collected a certain amount of training data, you can click the button to start training. The purpose of training is to allow the model to correctly recognize pictures representing four different directions.

Inference（Play game）

After the training is complete, you can play the game right away! You need to make different gestures in front of the camera in real time to control the direction of Pac-Man. This is how the game operates. Essentially, it is the process of model reasoning. For example, in the picture below, I made an upward gesture, and the model thought that my gesture was upward through reasoning (the reasoning was correct at this time), and then it controlled Pac-Man to go upward.

The above is a brief introduction to TensorFlow.js! Do you find it interesting? If you are interested, you can visit here for more detailed information.