Skip to main content

UI in XR

In an immersive session there is no flat screen to put a HUD on: the user sees the scene through a headset, or through a phone's camera in AR. Interfaces in XR are world-space screens, panels in the scene that the user points at with controllers or hands. Screen-space screens are made for flat displays, so don't use them in XR.

XR UI

Building a Panel​

Build a panel as you would any world-space screen, and size it in meters. At a scale of 0.001, one unit is a millimeter: a screen with a resolution of 800 × 500 is 80 cm wide, and text with a font size of 32 is 3.2 cm tall. Check the result in a headset, as text that reads well on a monitor can be too small in XR.

Where the panel lives decides how it moves:

  • In the world. A panel at a fixed place in the room, such as a control panel on a table, is the most comfortable. Place it about an arm's length to two meters from the user, a little below eye level, and turned toward them.
  • On a hand or controller. A panel that is a child of a controller's entity moves with it, like a wrist menu. The XrMenu script does this for hands and controllers.
  • Following the view. A panel fixed to the camera stays in the same place in view whatever the user does, which many people find uncomfortable. Let it follow the view slowly instead, as XrMenu's always-visible mode does.

Pointing and Selecting​

XR input goes through the application's XR manager, app.xr. The Editor, React and Web Components create it for you. An application created with pc.AppBase has one only if you add options.xr = pc.XrManager to its AppOptions.

While a session runs, the ElementInput casts the ray of every XR input source, each controller or hand, at the elements that have input enabled, every frame. It fires these events on the elements:

EventFired when
selectenter, selectleaveA ray moves onto or off the element
selectstartA select, such as a trigger press or a pinch, starts while the ray is on the element
selectmoveEvery frame while a select that started on the element continues
selectendThe select ends
clickThe select started and ended on the element

So a button is hovered while a ray points at it, pressed while a select holds it, and clicked when the select ends over it, just as with the mouse. Its click listeners need no changes for XR. See Input Sources for what a select is on each kind of input source.

Two properties of an input source control its interaction with the interface:

app.xr.input.on('add', (inputSource) => {
    // Let only the right-hand controller or hand point at the interface
    if (inputSource.handedness !== pc.XRHAND_RIGHT) {
        inputSource.elementInput = false;
    }
});

app.on('update', () => {
    for (const inputSource of app.xr.input.inputSources) {
        // The entity of the element that this input source points at, or null
        const target = inputSource.elementEntity;
        if (target) {
            // for example, shorten the drawn laser to end at the panel
        }
    }
});

To keep XR input away from all elements, create the ElementInput with useXr: false.

Hand Tracking​

With hand tracking, each hand is an input source with a ray of its own, and a pinch of the thumb and index finger is its select. Elements and buttons respond to hands as they do to controllers. See Hand Tracking.

XR Menus​

The engine's XrMenu script builds a menu of buttons from a list, and shows it on the palm of an open hand turned toward the user, on a controller when a button on it is pressed, or always, following the view. Each item fires an application event when it is selected. It picks the menu's buttons with its own ray and fingertip tests, so it works with the controllers' rays and with a fingertip poking a button.

import { XrMenu } from 'playcanvas/scripts/esm/xr/xr-menu.mjs';

const menu = new pc.Entity('menu');
menu.addComponent('script');
menu.script.create(XrMenu, {
    properties: {
        menuItems: [
            { label: 'Restart', eventName: 'menu:restart' },
            { label: 'Exit', eventName: 'xr:end' }
        ],
        fontAsset: font
    }
});
app.root.addChild(menu);

app.on('menu:restart', () => {
    restartLevel();
});

An item with a label but no eventName is a line of text rather than a button, and setItemLabel() changes an item's text while the menu is shown. The xr:end event in this example ends the session when the engine's XrSession script is in the scene, as it handles that event; see XR Scripts for what it does. XrMenu also fires xr:menu:active when the menu is shown or hidden.

XR Menu

Comfort​

  • Keep panels still, or let them follow slowly. Don't fix them rigidly to the view.
  • Put the controls that are used most within easy reach and view, so that users don't need to stretch or turn.
  • Make buttons large, and give small ones hit padding: pointing a ray precisely is harder than moving a mouse.
  • Show what the ray points at, for example with the hover tint of a button.

Entering XR​

Browsers only start an immersive session in response to a user action, such as a click or a tap. The engine's XR examples start it from an HTML button over the canvas. See Using WebXR for how to start and end a session.

See Also​

  • World-Space UI - Building and placing world-space screens
  • Input Sources - Controllers, hands and their rays and selects
  • Buttons - Button states, events and hit padding
  • XR - Immersive VR and AR with PlayCanvas