Official
Interhaptics
Blog

Your go-to resource for all the latest info and guides.

April 8th, 2020  |  By Interhaptics

Hand tracking for extended reality: naturally inspired UX and input

After speaking of haptics design and Interaction Building, let’s talk about one of the newest and most powerful technology: hand tracking for VR/AR/MR and its input mechanisms.

Hand tracking for extended reality is bringing a new level of immersion and user experience into XR applications. Indeed, users see a button, reach out and click with their fingertips. Everything is easy there! Super user experience, little usage friction and excellent immersion.

This feature is so compelling that almost every major headset maker is implementing hand tracking for VR/AR/MR for their headsets.

headsets_using_handtracking_technology_interhaptics_png
Figure 1: Some of the most common headsets

Hand tracking for extended reality works undoubtedly well because the user doesn’t have to learn anything while interacting with a virtual object. The user sees an object and reacts exactly as they would in real life. A ball on the table? Reach out and grab it!

Hand tracking for VR/AR/MR: the GRASP Taxonomy

It can be argued that the development of a great intelligence as human species is tied to the ability to create and manipulate new tools and objects. When we look in details into how we, as humans, grasp and manipulate objects we come out with a map explained in The GRASP Taxonomy of Human Grasp Types 1 illustrated in Figure 2 .

T. Feix, J. Romero, H. Schmiedmayer, A. M. Dollar and D. Kragic, “The GRASP Taxonomy of Human Grasp Types,” in IEEE Transact
Figure 2 : T. Feix, J. Romero, H. Schmiedmayer, A. M. Dollar and D. Kragic, The GRASP Taxonomy of Human Grasp Types, in IEEE Transactions on Human-Machine Systems, vol. 46, no. 1, pp. 66–77, Feb. 2016.

This subdivision is split in two major categories: Power and Precision manipulation, as a function of how we control the position of the object. Moreover, finger pad is for precision manipulation and hand palm is related to power manipulation. This is fundamental for hand tracking for extended reality.

Indeed, what we see in this division is the radically different way how we manipulate different objects. More precisely, how the finger is arranged and how the pre-manipulation action is made to accommodate the final position. We implemented these aspect into hand tracking for VR/AR/MR, read the paragraphs below to discover how.

pinch_vs_grab_interhaptics_gif
Figure 3: Different ways to grab an object

Hand tracking for extended reality: our experience

During the last several years working on customers projects using hand tracking for VR/AR/MR, we realized that a unique pinching and grabbing input were not responding to the necessities of smooth user experience. On the contrary, they generated a significant amount of frustration in the final user. For instance, pinching to move a large object is not expected at all by the user!

In some cases, we ended up avoiding the immersive characteristic of the environment because of the inconsistency of the user’s input actions. Therefore, users were not able to use the product. That means that we had a non-consistent user expectation across the product. Some interactions were realistic, and some of them had a high level of abstraction. This choice was not driven by the user preference, but by an inconsistent response of the hand tracking for extended reality not taking into considerations human factors.

Natural Gesture Hand tracking for AR/VR/MR interaction

The mission of Interhaptics is to give consistent, realistic, and immersive development tools for XR creators to increase the value of the final product. For this reason, in partnership with several research experts we developed grabbing algorithms inspired to the human behaviour for hand tracking for VR/AR/MR. The objective of these algorithms is to give the best UX possible while using hand tracking for extended reality.

We segmented the real hand behaviour, extracted the core parameters, and transcribed it in virtual reality to create a consistent tool for hand tracking for VR/AR/MR! This will be available for any hand tracking software supported by Interhaptics: Oculus, Vive, Leap Motion and the other planned in the future.

In the next paragraphs you can see some examples:

Precision Grab

precision_grasp_interhaptics_gif

Power Grab with Thumb

power_grasp_with_thumb_interhaptics_gif

Power Grab without Thumb

power_grasp_without_thumb_interhaptics_gif

Extreme cases

extreme_cases_interhaptics_

As you can see, the hand tracking for extended reality is a fundamental aspect of it. For us, its development is the result of a long reflection to improve each characteristics of its features. By observing the way each of us grabs of pulls something, we built the most realistic technology in order to maintain and optimize your own XR use. But this process is still open to your feedback, which enriches the development.

Click here to see our entire video dedicated to hand tracking for VR/AR/MR and subscribe to our channel for more content. These features will be shortly available in the upcoming version of Interhaptics.

Check out all our articles here to read about how haptics keep you immersed into your VR experiences. Extend your reality now by downloading Interhaptics and design, develop and deploy your own interactions.

Comments

3 responses to “Hand tracking for extended reality: naturally inspired UX and input”

  1. […] our last blog post, we explained how to create 3D interaction for XR. Now it’s time to understand how to use it. As a reminder, Interhaptics is a cross-platform […]

  2. […] Each interaction is triggered logically and the behavior of the object in the 3D space is also logically managed therefore you just need to apply script and the object will behave as it should. A small detail that is actually important in terms of user experience is the differentiation between big and small objects. It was a necessity since we do not grab objects the same way depending on their size (have more details about different ways to grab here). […]

  3. […] Each interaction is triggered logically and the behavior of the object in the 3D space is also logically managed therefore you just need to apply script and the object will behave as it should.A small detail that is actually important in terms of user experience is the differentiation between big and small objects. It was a necessity since we do not grab objects the same way depending on their size (have more details about different ways to grab here). […]