As virtual and augmented reality steadily advance in both visual fidelity and headset comfort, researchers continue to work on input solutions that will feel more natural than holding controllers. A group of researchers announced FingerTrak, a wristband-based solution that uses thermal cameras to track hand movements in 3D, abstracting 20 finger joint positions from contours on the wearer’s wrist.
Developed by Cornell University’s SciFi Lab with assistance from University of Wisconsin, Madison researchers, FingerTrak uses a deep neural network to stitch together input from three or four miniature thermal cameras mounted around the wrist, collectively capturing an entire hand pose. Using silhouettes generated by the cameras, backbone and regression networks estimate fingertip and joint positions, and though the results aren’t perfect, they could be used for some forms of VR and AR input. Other potential applications for FingerTrak include human-robot interaction and control, sign language translation, and mobile health, including early detection of degenerative diseases such as Parkinson’s and Alzheimer’s.
Interestingly, the researchers suggest that wrist contours alone are “enough to accurately predict the entire hand posture,” enabling the entire sensing system to be placed on the wrist, rather than requiring gloves, rings, or other techniques that have previously been released. A demonstration video shows FingerTrak’s hand motion tracking translating into movements for a bionic hand, as well as enabling a computer to determine when a user is writing, drinking coffee, and interacting with a smartphone.
That said, it’s unclear whether the system will be able to rapidly track specific gestures, such as what a person is writing; the researchers note that FingerTrak’s average angular error rate ranged from 6.46 to 8.06 degrees during testing, depending on the background it was tested against. In other words, it might best supplement rather than fully replace higher accuracy finger tracking solutions, at least for some applications.
Wrist-mounted cameras could augment the inside-out tracking cameras found in VR headsets such as Oculus Quest, which monitor hand and finger positions from the user’s head. In prototype form, FingerTrak is already pretty small, and it could easily go smaller with further engineering. It relies chiefly on very low-resolution (32- by 24-pixel) thermal cameras that are pea-sized and thinner than the thinnest Apple Watch, albeit with the need to be mounted at multiple points on a wristband. The prototype also relies on a tethered Raspberry Pi board, though the hardware could alternately be controlled through a smaller solution, such as a smartwatch.
FingerTrak will be presented in mid-September at the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing. The underlying paper, “FingerTrak: Continuous 3D Hand Pose Tracking by Deep Learning Hand Silhouettes Captured by Miniature Thermal Cameras on Wrist,” is available here.
This post by Jeremy Horwitz originally appeared on VentureBeat.
The post Researchers Show FingerTrak, A Hand Tracking Wristband For AR/VR Input appeared first on UploadVR.
from UploadVR https://ift.tt/3g337De
via IFTTT
No comments:
Post a Comment