Skip to main content

Mesh Detection

Interaction between real-world and virtual objects is achieved via visual and logical interactions between the two. Mesh detection is an API that provides access to the representation of real-world geometry in the form of meshes. It can be used in a number of ways such as:

  • Virtual object physics within a real-world environment
  • Path finding
  • Object placement
  • Occlusion
  • Procedural effects

This API provides a list of meshes, their geometry, transformation and semantic labeling.

The underlying system can provide pre-captured data as well as provide real-time reconstruction depending on the underlying system implementation.

Support​

if (app.xr.meshDetection.supported) {
    // mesh detection is supported
}

app.xr.on('start', () => {
    if (app.xr.meshDetection.available) {
        // mesh detection is available
    }
});

Access​

A feature flag needs to be added to the session start:

app.xr.start(camera, pc.XRTYPE_AR, pc.XRSPACE_LOCALFLOOR, {
    meshDetection: true
});

Meshes are added/removed asynchronously:

app.xr.meshDetection.on('add', (xrMesh) => {
    // a new XrMesh has been added

    xrMesh.once('remove', () => {
        // an XrMesh has been removed
    });
});

Also, the list of XrMeshes is available:

const xrMeshes = app.xr.meshDetection.meshes;
for (let i = 0; i < xrMeshes.length; i++) {
    const xrMesh = xrMeshes[i];
}

Mesh​

Each mesh is represented as an instance of XrMesh, which can be added/removed during an active session. It also has data that can be changed during its lifetime.

You can access the position and rotation (world-space) of an XrMesh:

entity.setPosition(xrMesh.getPosition());
entity.setRotation(xrMesh.getRotation());

Each XrMesh has its vertices and indices (in local-space), that can be used to construct a visual mesh. An example below creates a visual mesh for each XrMesh and adds it to the root of the scene:

app.xr.meshDetection.on('add', (xrMesh) => {
    // geometry data
    const mesh = new pc.Mesh(app.graphicsDevice);
    mesh.clear(true, true); // ensure that mesh is created with dynamic buffers
    mesh.setPositions(xrMesh.vertices); // set vertices
    mesh.setNormals(pc.calculateNormals(xrMesh.vertices, xrMesh.indices)); // calculate normals
    mesh.setIndices(xrMesh.indices); // set indices
    mesh.update(pc.PRIMITIVE_TRIANGLES); // update buffers

    const material = new pc.StandardMaterial();
    const meshInstance = new pc.MeshInstance(mesh, material);

    const entity = new pc.Entity();

    // add render component
    entity.addComponent('render', {
        meshInstances: [ meshInstance ]
    });

    // add entity to the scene root
    app.root.addChild(entity);

    // clean up after XrMesh is removed
    xrMesh.once('remove', () => {
        material.destroy();
        mesh.destroy();
        entity.destroy();
    });
});

Semantic Label​

XrMesh can represent various real-world objects and a label can help to identify what it represents using its property XrMesh.label.

These labels can be any of: floor, wall, door, window, table, screen, global mesh, other, and more. Here is a list of semantic labels, although this list is not definitive and the platform can report anything it feels fit.

Changes​

Depending on the underlying system capabilities, the XrMesh geometry can change while an XR session is active. You can subscribe to that event and update a visual mesh accordingly:

xrMesh.on('change', () => {
    // vertices, indices and/or label has been changed
});