The phenomenon where the tip of a finger turns white when pressed against a hard surface has become the foundation for a new technology that enables augmented reality (AR) panels to be placed on real-world objects such as walls and tables.

There are existing methods that allow sensor panels and keyboards to be projected so they appear to float in front of users wearing AR headsets. However, interacting with these interfaces typically requires special controllers or gloves equipped with sensors. Additionally, holding your hands up for extended periods can cause fatigue, and the lack of physical contact with a real surface reduces tactile feedback.

The new experimental system uses a standard headset camera to detect the whitening of the fingertip when it presses against a solid surface. Artificial intelligence analyzes this effect to determine the exact moment and location of contact between the finger and the virtual sensor panel projected onto the chosen surface. Once contact is detected, the AR software executes the corresponding action.

This approach allows any surrounding surface—walls, tables, partitions—to be used as a sensor panel without the need for additional sensors, markers, or devices. The technology does not require special equipment and can be used by anyone.

The results of this work were presented at the 33rd IEEE Conference on Virtual Reality and 3D User Interfaces, which took place last month in South Korea.