Hello readers!

Remember when we talked about the virtual lessons in Metaverse? Well, I recently got to see that potential in action at a special event, and let me tell you, it was mind-blowing! So, grab a seat and let me take you on a journey through a day filled with innovation and inspiration.

First, we visited a classroom where the teacher was educating students about the dangers that could occur at various spots throughout the trip on foot from Aizuwakamatsu station to the Support School.

The lecture materials were mostly 360 videos recorded by one of the teachers. Several scenarios were presented to the students, and they were asked to identify the potential dangers in each situation. Each student, one by one, could also experience the scenario firsthand by wearing an Oculus Quest 3 for an immersive experience, making it easier for them to understand. Subsequently, they were briefed about the dangers by “Ai-chan,” a virtual mascot animated using mocopi (a motion capture software) and voiced by one of the school’s teachers.

There was a particular scenario that involved students walking through a bus stop on a narrow sidewalk during while it was raining. In this case, three patterns were presented to the students:

1. Walking through the bus stop while holding an umbrella, while another person is coming from the opposite direction also holding an umbrella. Naturally, both people cannot fit through the bus stop, so what are the proper manners to handle such a situation? Answer: The student should fold their umbrella slightly so that both the oncoming person and themselves can comfortably pass through the bus stop.
2. Walking through the bus stop while there is an oncoming person pushing a bicycle by their side. Answer: To slightly turn sideways, pause for a moment, and let the person with the bicycle pass through.
3. Passing through the bus stop while another person is walking towards you from the opposite direction. Naturally, the answer is to slightly move to the side so that both the student and the oncoming person can pass through.

These types of scenarios might not be clear to every student at the Special School due to their mental or physical disabilities, so it is important to prepare them for real-life situations that could be challenging to navigate spontaneously.

Hence, there were a couple of props not only to discuss the three scenarios above but also to re-enact them. So, while the student was exploring those scenarios, they not only experienced being at the bus stop thanks to VR but also had physical representations of the bus stop, such as a small bench and a bus stop sign in the classroom, to practice proper responses. The teacher then simply asked, “What would you do if there were a person with an umbrella walking in the opposite direction?” One of the students immediately answered, “To fold the umbrella and create more space for both people to pass.” Everyone was surprised by this answer, as it was perfectly explained and demonstrated. Similarly, the rest of the students found answers to the other scenarios together as well.

To properly memorize those newly found answers about daily life, one of the students was selected to use mocopi to record those correct answers. He put on a couple of sensors that mocopi uses, and thanks to motion capture, he could become a 3D avatar that he chose to be. Looking at his own avatar, mirroring his movements, he narrated the correct solutions in a couple of sentences, using proper gestures to accompany the narration. This narration and motion-captured video of the student’s 3D avatar will then be used to accompany the above-mentioned “Ai-chan” in an updated instructional video about the hazards and proper mannerisms on the way from Aizuwakamatsu station to the Support School. Essentially, students, alongside teachers, becoming content creators for their fellow students, creating interactive, immersive, and instructional videos about the dangers and proper behaviour in Japanese society.

After such a brilliant class, we were then greeted by the students, and they even prepared hand-written thank-you letters. I really had to hold back the tears of joy and happiness after hearing and seeing how much they enjoy using these technologies to learn about various things so their lives can be easier in the future.

I truly believe that this program is great for them to overcome several difficulties they might face, one of them being the lack of confidence. According to the school’s principal, these content creation activities tremendously help students not only better understand the world around them but also to better understand themselves. When they talk and record short talks about their newly acquired knowledge, they have the opportunity to see themselves in a character form that they like, possibly admire, and can therefore observe their movements and reactions, rather than just recording a video without any feedback for themselves. And by having this sort of mirror in front of them, reflecting something they are comfortable with, yet still reflecting their behaviour, helps with their confidence during presentations.

Continuing to another class, where students were learning about how to use buses in the city of Aizuwakamatsu. This class brought me back to the days when I moved to Aizuwakamatsu and took the city bus for the first time.

Surely, the system of using buses isn’t particularly strange, difficult, or nonsensical in any way, but it is a system that one has to get used to. Perhaps to a person who grew up around these parts of Japan, it might come naturally via teachings of their parents, but to students with disabilities, much like to foreigners, such a system might not be as easy as saying, “I see, that is how it is,” so to speak.

Needless to say, this class brought back a lot of memories, especially when I was trying to figure out the ropes of Japanese countryside public transportation, where it isn’t as simple as just “beep your Suica here”.

Yet the whole time during this class, I was thinking, ‘I wish I had 360 videos of the inside of our city bus and an explanation of how things work here, there, and what is what. This is absolutely brilliant!’. Hence from the perspective of a person without any disability, this kind of lesson would have made my life way easier about 4-5 years ago as well.

After this class, our CEO, Yamadera-san, an intern named Felix, and I joined a presentation showcasing the achievements of the teachers and students during the last academic year of 2023. The presentation featured the technologies I mentioned earlier, including the Meta Quest 3, mocopi, binaural microphone, and 360 videos.

It was also heartening to see that some of the videos featured the school’s mascot, Power-kun, whose 3D version was designed by my colleague (and fiancée), Julia. The mascot can also be utilized in mocopi as an avatar. Additionally, we will soon present an app that will allow teachers to incorporate simple animations of Power-kun into their instructional videos.

(see Power-kun in the featured image of this post)

Overall, the entire experience of today’s event was immensely heartwarming!

Witnessing students and teachers enjoying the technologies introduced by Eyes, JAPAN not long ago, and seeing how they’ve ingeniously applied them to prepare students for better and easier lives, is truly beautiful.

As someone who aspires to become a teacher or professor again in my life, today’s experience has truly evoked heartwarming feelings and motivation to continue working on projects that benefit people.

I extend my heartfelt thanks to everyone who was part of this project, especially the Fukushima Prefectural Aizu Support School and their brilliant staff of teachers.

I look forward to continuing to be part of this project in the future!