メインコンテンツまでスキップ

Mesh Detection

Interaction between real-world and virtual objects is achieved via visual and logical interactions between the two. Mesh detection is an API that provides access to the representation of real-world geometry in the form of meshes. It can be used in a number of ways such as:

  • Virtual object physics within a real-world environment
  • Path finding
  • Object placement
  • Occlusion
  • Procedural effects

This API provides a list of meshes, their geometry, transformation and semantic labeling.

The underlying system can provide pre-captured data as well as provide real-time reconstruction depending on the underlying system implementation.

Support

if (app.xr.meshDetection.supported) {
// mesh detection is supported
}

app.xr.on('start', () => {
if (app.xr.meshDetection.available) {
// mesh detection is available
}
});

Access

A feature flag needs to be added to the session start:

app.xr.start(camera, pc.XRTYPE_AR, pc.XRSPACE_LOCALFLOOR, {
meshDetection: true
});

Meshes are added/removed asynchronously:

app.xr.meshDetection.on('add', (xrMesh) => {
// a new XrMesh has been added

xrMesh.once('remove', () => {
// an XrMesh has been removed
});
});

Also, the list of XrMeshes is available:

const xrMeshes = app.xr.meshDetection.meshes;
for (let i = 0; i < xrMeshes.length; i++) {
const xrMesh = xrMeshes[i];
}

Mesh

Each mesh is represented as an instance of XrMesh, which can be added/removed during an active session. It also has data that can be changed during its lifetime.

You can access the position and rotation (world-space) of an XrMesh:

entity.setPosition(xrMesh.getPosition());
entity.setRotation(xrMesh.getRotation());

Each XrMesh has its vertices and indices (in local-space), that can be used to construct a visual mesh. An example below creates a visual mesh for each XrMesh and adds it to the root of the scene:

app.xr.meshDetection.on('add', (xrMesh) => {
// geometry data
const mesh = new pc.Mesh(app.graphicsDevice);
mesh.clear(true, true); // ensure that mesh is created with dynamic buffers
mesh.setPositions(xrMesh.vertices); // set vertices
mesh.setNormals(pc.calculateNormals(xrMesh.vertices, xrMesh.indices)); // calculate normals
mesh.setIndices(xrMesh.indices); // set indices
mesh.update(pc.PRIMITIVE_TRIANGLES); // update buffers

const material = new pc.StandardMaterial();
const meshInstance = new pc.MeshInstance(mesh, material);

const entity = new pc.Entity();

// add render component
entity.addComponent('render', {
meshInstances: [ meshInstance ]
});

// add entity to the scene root
app.root.addChild(entity);

// clean up after XrMesh is removed
xrMesh.once('remove', () => {
material.destroy();
mesh.destroy();
entity.destroy();
});
});

Semantic Label

XrMesh can represent various real-world objects and a label can help to identify what it represents using its property XrMesh.label.

These labels can be any of: floor, wall, door, window, table, screen, global mesh, other, and more. Here is a list of semantic labels, although this list is not definitive and the platform can report anything it feels fit.

Changes

Depending on the underlying system capabilities, the XrMesh geometry can change while an XR session is active. You can subscribe to that event and update a visual mesh accordingly:

xrMesh.on('change', () => {
// vertices, indices and/or label has been changed
});