First look at Sony’s finger-tracking VR controllers

Sony is working on a pair of finger-tracking VR controllers.
Image: UploadVR

The world of virtual reality (VR) is in a far better place than it was even five years ago. Thanks to affordable hardware and high-quality applications, more consumers than ever are using it. However, without further innovation, VR cannot break into the mainstream.

Sony is working on a project that would certainly have a positive impact on the future of VR. For the past two years, patent filings have made it known that the company is working on finger-tracking technology for a line of VR motion controllers. Recently, UploadVR spotted a research paper and a demo video from two Sony researchers that gives us a first look at the tech.

Watching Your Fingers

Making VR more interactive is a top priority for both hardware and software makers. Watching a movie in virtual reality or immersing yourself in a 3D scene is fun. Being able to naturally interact with the environment takes the experience to another level.

Manage your supply chain from home with Sourcengine

Although there are various types of VR controllers out there, they don’t replicate the movements of a user’s hand. Sony’s new finger-tracking controllers are designed to do just that.

Researchers Kazuyuki Arimatsu and Hideki Mori published a paper titled “Evaluation of Machine Learning Techniques for Hand Pose Estimation on Handheld Device with Proximity Sensor.” They detail how their prototype uses capacitive proximity sensors to determine where a user has their hands. Moreover, the sensors detect the movements of each finger individually. These readings are then replicated in real-time in virtual reality.

The user sees a rendered pair of hands that mimic what their own hands do in real life.

The research team behind the technology built training datasets by capturing images of the hands of 12 people—all of varying sizes—using an optical tracking system. Their approach uses a controller that is strapped to the back of a user’s hand. In a video, the researchers showed off the controllers in action as a user stacked virtual blocks and posed a digital model while wearing them.

Unanswered Questions

While the finger-tracking controllers certainly look awesome, there are still a lot of questions surrounding them. For one, are they being designed for a second-generation PlayStation VR system?

It’s worth noting that Arimatsu and Mori are part of Sony Interactive Entertainment—the division behind PlayStation’s development. As of now, nothing has been confirmed. However, it wouldn’t be surprising to see some version of the controllers as part of the PlayStation lineup down the road.

Regardless, finger-tracking technology has implications in the wider VR space as well. Combining it with something like haptic touch, for instance, would allow users to not only see their hands in VR but actually feel the environment as they interact with it. Bringing together sight and touch would greatly enhance the immersiveness of VR.

Even without haptic technology, finger-tracking controllers have the potential to revolutionize the VR world. Sony isn’t the only company working on the tech, but seeing it in action is a good reason to get excited about its future.


Please enter your comment!
Please enter your name here