Skip to main content

@realitycollective/babylon-interactions

The Babylon.js adapter for the Reality Collective Interaction Extensions. It feeds a Babylon WebXR experience into the @realitycollective/webxr-interactions core.

npm install @realitycollective/babylon-interactions

It re-exports everything from the core, so this is the only interaction package your app needs.

Not yet exercised against a real Babylon runtime. The adapter is written against the documented Babylon 7 API and covered by structural fakes. Treat it as a preview until it has run in a Babylon app.

What it binds​

LayerDetail
InputWebXRDefaultExperience: controllers, motion controller components (trigger, squeeze), hand-tracking joints, and the session manager for what is live
Hit-testingA sphere test over registered nodes, or your own scene.pickWithRay through the pickWithRay hook
MovementMoves, rotates and scales Babylon nodes for grab, hinge, dial and slide
Desktopscene.onPointerObservable as a pointer fallback, so the same scene is testable without a headset
HapticsThrough the motion controller's pulse
No @babylonjs/core dependencyIt matches the shape of the Babylon API in TypeScript rather than importing Babylon, so an upstream release cannot break your install

Usage​

import { createBabylonInteractions } from "@realitycollective/babylon-interactions";

const xr = await scene.createDefaultXRExperienceAsync();
const interactions = createBabylonInteractions({ scene, xr, attachToScene: true });

interactions.register({ id: "button", behaviours: [{ kind: "press" }] }, buttonMesh);
interactions.runtime.onEvent((event) => console.log(event.type));

attachToScene drives the update loop from scene.onBeforeRenderObservable and the engine's frame delta. Leave it off and call interactions.update(dtSeconds) from your own loop.

For mesh-accurate targeting, hand the adapter your own pick:

interactions.setPickWithRay((origin, direction, maxDistance) => {
  const info = scene.pickWithRay(new Ray(Vector3.FromArray(origin), Vector3.FromArray(direction), maxDistance));
  return info?.hit && info.pickedPoint
    ? { mesh: info.pickedMesh, distance: info.distance, point: info.pickedPoint.asArray() as [number, number, number] }
    : null;
});

Things to know​

  • Forward is +Z. Babylon is left-handed, so a controller ray points down +Z where three.js and raw WebXR point down -Z. A scene that sets useRightHandedSystem faces -Z instead; the adapter reads that flag at construction and flips. The adapter handles this; it matters if you compare rays with another adapter's.
  • Rotations need a quaternion. A node whose rotationQuaternion is null is still driven by Euler angles. Set node.rotationQuaternion = Quaternion.Identity() before registering it, or pass createQuaternion to register, otherwise the first rotation write stores a plain object that Babylon cannot use.
  • Presence shows and hides what Babylon built: motion controller root meshes and hand meshes. Babylon picks the visual per input source, so there is no hands/controllers switch - setPresenceModality always returns false.
  • Desktop grip. The pointer fallback puts its grip one metre along the pointer ray, matching the three.js adapter, so grab, hinge, dial and slide follow the cursor on desktop. Set desktopGripDistance near the distance of the things being manipulated; at 0 the grip sits on the camera and a drag reports camera motion only.

Peer dependency​

None. Babylon is matched structurally, not imported, so any Babylon version whose objects carry these members works.

Live demo​

The interaction playground - the three.js build of the same station set: webxr-interactions.pages.dev

Documentation​

See the repository README.

License​

MIT - see LICENSE.

Classes​

ClassDescription
BabylonHitTesterResolves which interactable a ray or a proximity probe touches. Implemented by the adapter (three.js Raycaster, engine BVH, …). A provider that pre-resolves targeting (IWSDK) may make this redundant - provider hints always take precedence over hit-tester results.
BabylonInputProvider-
BabylonInteractions-
BabylonTransformPortThe transform/effect surface of ONE interactable's scene object. Behaviours write through this; the adapter owns how writes land (Object3D fields, ECS components, …).
DialBehaviour-
DwellState-
EmitterMinimal synchronous emitter (insertion-ordered, unsubscribe-safe).
GrabBehaviour-
HingeBehaviour-
InteractionRuntime-
PressBehaviour-
PulseBehaviour-
SlideBehaviour-
TossScoreBehaviour-
VelocityTracker-

Interfaces​

InterfaceDescription
BabylonCameraLikeStructural slice of Babylon's Camera.
BabylonEngineLikeStructural slice of Babylon's Engine - only the frame delta is read.
BabylonHandTrackingLikeStructural slice of WebXRHandTracking, the feature the features manager registers under "xr-hand-tracking".
BabylonHitTesterOptions-
BabylonInteractionsOptions-
BabylonMotionControllerComponentLikeStructural slice of one WebXRControllerComponent reading.
BabylonMotionControllerLikeStructural slice of Babylon's WebXRAbstractMotionController.
BabylonObservableLikeStructural slice of Babylon's Observable<T>. The observer handle is opaque - it is only ever handed straight back to remove.
BabylonPickingInfoLikeStructural slice of Babylon's PickingInfo.
BabylonPickResultWhat an app's mesh-accurate pick reports back.
BabylonPointerInfoLikeStructural slice of Babylon's PointerInfo. type is one of the PointerEventTypes constants - see POINTER_EVENT_TYPES.
BabylonProviderOptions-
BabylonQuaternionLikeStructural slice of Babylon's Quaternion.
BabylonRayLikeStructural slice of Babylon's Ray.
BabylonRegisterOptions-
BabylonSceneLikeStructural slice of Babylon's Scene.
BabylonTransformNodeLikeStructural slice of Babylon's TransformNode, plus the one member AbstractMesh adds that this adapter reads (isVisible).
BabylonTransformPortOptions-
BabylonVector3LikeStructural slice of Babylon's Vector3.
BabylonXRControllerLikeStructural slice of Babylon's WebXRInputSource.
BabylonXRExperienceLikeStructural slice of Babylon's WebXRDefaultExperience.
BabylonXRHandLikeStructural slice of one tracked hand from the hand-tracking feature.
BabylonXRInputLikeStructural slice of Babylon's WebXRInput.
Behaviour-
BehaviourContextEverything a behaviour can reach during a tick or a routed moment.
BehaviourFactoryContext-
DialConfig-
DwellConfigGaze support - first-class, per the design decision that gaze serves two roles:
DwellTick-
FeedbackAudioSinkWhatever the client uses to play a sound. Deliberately tiny: an id the client resolved to a buffer/sample of its own, and a gain. The core never loads, decodes or owns audio.
FeedbackIntent-
GazeConfig-
GrabConfig-
HingeConfig-
HitTesterResolves which interactable a ray or a proximity probe touches. Implemented by the adapter (three.js Raycaster, engine BVH, …). A provider that pre-resolves targeting (IWSDK) may make this redundant - provider hints always take precedence over hit-tester results.
InteractableDescriptor-
InteractableHit-
InteractableState-
InteractionDescriptor-
InteractionEvent-
InteractionRuntimeOptions-
InteractorInfoThe slice of a live interactor a behaviour may consume.
PointerBridge-
PressConfig-
PulseConfig-
RegisterPorts-
SlideConfig-
TossScoreConfig-
TransformPortThe transform/effect surface of ONE interactable's scene object. Behaviours write through this; the adapter owns how writes land (Object3D fields, ECS components, …).
VelocityTrackerOptions-

Type Aliases​

Type AliasDescription
BabylonPickWithRayA mesh-accurate pick supplied by the app - scene.pickWithRay with a Ray built from these arguments. Return null for a miss.
BehaviourConfig-
FeedbackCueA named moment a client may want to sonify/hapticise.
FeedbackListener-
GrabFulfilment-
InteractionEventListener-
InteractionEventType-
TransformOwnershipWho owns the interactable's transform while this behaviour is active.

Variables​

VariableDescription
DESKTOP_SOURCE_IDThe desktop pointer source's id - stable for the life of the page.
DWELL_DEFAULTS-
HAND_TRACKING_FEATUREThe name Babylon registers hand tracking under in the features manager.
HEAD_GAZE_INTERACTOR_IDInteractor id used when gaze is synthesized from the head pose.
INDEX_TIP_JOINTIndex fingertip joint, as WebXR and Babylon both spell it.
POINTER_EVENT_TYPESThe PointerEventTypes values this adapter reacts to. Babylon defines them as one bit per event; these three are unchanged across 5, 6 and 7.

Functions​

FunctionDescription
centeredUnitMap a centered hinge angle in `[-
clamp01Clamp to the unit interval.
clampDeadzoneZero out a signal below threshold, leaving larger values untouched.
clampSymClamp a value to the symmetric range `[-
createBabylonInteractions-
createBehaviour-
createPointerBridgeCreate a pointer stream from the runtime's sampled sources. match picks which source drives it (default: any source with a ray - first pressed wins and holds until release).
decayExponential decay of value toward 0 over dt at ratePerSec.
defaultForwardThe world-space forward axis for a scene: +Z in Babylon's default left-handed system, the opposite of three.js and of a raw WebXR target ray, and -Z when the scene sets useRightHandedSystem.
easeOutBouncePenner's ease-out-bounce on t ∈ [0,1].
hingeAngleToHandHinge angle for a position-driven lever: given the hand's offset from the pivot resolved into the hinge plane - alongRest (toward the arm's rest direction) and alongSwing (the in-plane perpendicular) - the angle the arm should take to point at the hand, clamped to `±
holderPointHand position a hand-driven behaviour should track, best available.
nodeForwardA node's forward direction in world space, honouring the scene's handedness (see defaultForward).
nodeShowingIs this node currently showing? A node the app disabled or hid is not a hit-test candidate. Absent members mean yes - a fake, or a build that does not carry them, should not silently drop out of targeting.
nodeWorldPoseA node's world pose: absolute position and absolute rotation.
normalize01Normalize a measured displacement/angle against a signed range to 0..1. Sign-agnostic: a range of -π/2 maps -π/2 → 1.
parentOfThe parent of a node, when it is one this adapter can read a world pose from. Babylon types parent as Node, which has no transform, so a parent that is a bone or a bare node reports null and the caller treats the node as unparented.
passedThroughHoopDid a tossed object pass through a hoop this frame? True when it crossed the hoop plane in the scoring direction (downward - prev above, curr at/below) while inside the ring (radial <= radius).
quatConjugate-
quatFromAxisAngleQuaternion from an axis (normalised) and an angle in radians.
quatMultiply-
rayAngleToAngular offset (radians) between a ray direction and the direction to a point.
rayPointDistanceClosest-approach distance from a ray to a point, and the along-ray t.
registerBehaviourKind-
registerDescriptorBuild a runtime's interactables from a portable descriptor.
resolveDwellConfig-
routeAudioToSinkClient opt-in: play a sound for the feedback intents named in cueMap. Cues absent from the map are ignored, so an app sonifies only the moments it has sounds for. Gain is the intent's intensity scaled by gainScale, capped at 1. Returns the unsubscribe.
routeHapticsToProviderClient opt-in: forward the haptic half of feedback intents to the input provider's pulse (which no-ops on sources without actuators - hands). Returns the unsubscribe. Audio remains entirely the client's concern.
springStepOne step of a damped harmonic oscillator (semi-implicit Euler). out[0] = next value, out[1] = next velocity. An under-damped spring (damping ratio ≈ 0.3-0.5) is what gives a pressed button its bounce. dt is clamped to 1/30 s so a long frame can't blow the integrator up.
toQuatCopy a Babylon quaternion into a tuple, defaulting to identity.
toVec3Copy a Babylon vector into a tuple.
vAdd-
vApplyQuatRotate a vector by a quaternion.
vCross-
vDistance-
vDot-
vLength-
vNormalize-
vScale-
vSub-
worldToLocalTransform a world-space point into the local frame of pose.
writeQuatWrite a tuple into an existing Babylon quaternion, in place.
writeVec3Write a tuple into an existing Babylon vector, in place.