Smartphone app can track finger movements

With the advent of smartwatches and smaller touchscreen devices, many people, especially those with limited dexterity, find it difficult to manipulate items on such small screens using just their fingers. In response to this, computer scientists and engineers at the University of Washington (UW) have developed an application, called FingerIO, that allows users to control their devices by tracking the movement of their fingers. Moving your finger on a table or on a piece of paper, or simply moving it in mid-air, would allow you to scroll through menus, press buttons or even reply to a text.

FingerIO functions by turning the smartphone or watch into an active sonar system by commandeering the use of its microphones and speakers. An audible sound wave is emitted from the microphone, and when the wave bounces off the user’s finger, the returning echoes are read by the speaker in order to calculate the finger’s location.

“You can’t type very easily onto a smartwatch display, so we wanted to transform a desk or any area around a device into an input surface,” lead author Rajalakshmi Nandakumar, a UW doctoral student in computer science and engineering, said in a press release. “I don’t need to instrument my fingers with any other sensors — I just use my finger to write something on a desk or any other surface and the device can track it with high resolution.”

When designing FingerIO, the team considered other methods that could potentially track users’ finger movements. Video, for example, was one consideration. However, this would require that the device possess a direct line of sight with the person’s hand. Radar was also considered, but this required custom sensor hardware and significant computing power. Working with sonar enables the app to function without any new hardware and allows the user to access the device when it is hidden by fabric, such as under a sleeve or in a pocket.

“Acoustic signals are great — because sound waves travel much slower than the radio waves used in radar, you don’t need as much processing bandwidth so everything is simpler,” Shyam Gollakota, senior author and UW assistant professor of computer science and engineering, said in a press release. “And from a cost perspective, almost every device has a speaker and microphones so you can achieve this without any special hardware.”

The biggest challenge the researchers faced regarding the sonar system was fine-tuning its precision. Sound wave echoes are naturally weak, making it difficult to precisely pinpoint a finger’s location at any given moment. When it comes to translating a user’s movements into commands, precision is necessary. An error of a few centimeters can mean the difference between entirely different letters when someone is typing. The app is able to track finger movements within eight millimeters, which is sufficient enough for modern smartphone devices.

The UW team’s success with FingerIO was revealed in a paper that is to be presented in May at the Association for Computing Machinery’s 2016 conference in San Jose, California.

To demonstrate the high resolution tracking of FingerIO, the team used Orthogonal Frequency Division Multiplexing (OFDM), a signal commonly used in wireless communication. Specially designed algorithms used the OFDM properties to track phase changes in echoes during several tests. Accounting for such changes enabled the device to automatically correct for errors in its detection of the finger location.

To test the app, the researchers downloaded the FingerIO prototype app onto a Samsung Galaxy S4 and a smartwatch. While the phone was simply an off-the-shelf model, the watch was customized with two microphones, an upgrade that was necessary for it to track in two dimensions. Test subjects then used their fingers to draw shapes on a touchpad while FingerIO tracked their movements. The shapes drawn on the pad were compared to the shapes computed by devices with FingerIO. The difference between the two averaged out to be only 0.8 centimeters for the phone and 1.2 centimeters for the smartwatch.

“Given that your finger is already a centimeter thick, that’s sufficient to accurately interact with the devices,” Vikram Iyer, an electrical engineering graduate student and a co-author of the paper, said in a press release.

Moving forward, the team plans to demonstrate that FingerIO can be used to track multiple fingers moving at the same time. They also hope to explore the extent of the app’s tracking abilities by adding additional microphones to the different devices.