Skip to main content

Hand Tracking

If the platform supports WebXR Hand Input, then an input source can have associated hand data, which is exposed as an XrHand, and its data in the form of XrFingers and XrJoints for an application developer to use, such as wrist, fingers, joints, tips and events for detecting when hands lose/restore tracking.


Creating a basic hand model:

const joints = [];
const hand = inputSource.hand;

if (hand) {
for (let i = 0; i < hand.joints.length; i++) {
const entity = new pc.Entity();
entity.joint = hand.joints[i];
entity.addComponent('render', { type: 'box' });


Every frame, joint data can change position, rotation, and other details.

for (let i = 0; i < joints.length; i++) {
const entity = joints[i];
const joint = entity.joint;
const radius = joint.radius * 2;
entity.setLocalScale(radius, radius, radius);


Hand tracking is subject to the reliability and sophistication of the underlying system. There might be cases when tracking is not possible due to obstructions between cameras and hands, or when hands interlock in a complex way. While Computer Vision techniques are improving, when designing content with hands as an input source, their shortcomings should be taken into consideration.