Fix WebXR body tracking startup hand roll (ring metacarpal twist reference)#18530
Fix WebXR body tracking startup hand roll (ring metacarpal twist reference)#18530RaananW wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Fixes a WebXR body-tracking startup issue where entering XR with a non-neutral hand pose (e.g. palm up) could bake a fixed wrist/forearm roll offset into the avatar for the rest of the session. The fix changes the first-frame bind capture to opt-in and instead always applies aim/twist correction in the direct retarget path.
Changes:
- Flip
autoCaptureBindOnFirstFramedefault fromtruetofalseand expose it as an option onIWebXRBodyTrackingOptions. - Apply unmapped aim-target and hand-plane twist correction in the direct retarget path (using
_boneAimTargetJointIdx/_boneTwistReferenceJointIdx) and cache per-frame world rotations across frames instead of reallocating the map each frame. - Add unit tests asserting the new default and that palm-up startup does not bias later neutral hand orientation.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| packages/dev/core/src/XR/features/WebXRBodyTracking.pure.ts | Changes default and option for auto bind capture; rewrites direct retarget path to apply aim + twist correction and reuse cached world-rotation map. |
| packages/dev/core/test/unit/XR/babylon.webXRBodyTracking.test.ts | Adds tests covering the new default and palm-up startup retarget behavior. |
|
Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). |
|
Snapshot stored with reference name: Test environment: To test a playground add it to the URL, for example: https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/18530/merge/index.html#WGZLGJ#4600 Links to test your changes to core in the published versions of the Babylon tools (does not contain changes you made to the tools themselves): https://playground.babylonjs.com/?snapshot=refs/pull/18530/merge To test the snapshot in the playground with a playground ID add it after the snapshot query string: https://playground.babylonjs.com/?snapshot=refs/pull/18530/merge#BCU1XR#0 If you made changes to the sandbox or playground in this PR, additional comments will be generated soon containing links to the dev versions of those tools. |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
|
Visualization tests for WebGPU |
|
WebGL2 visualization test reporter: |
🟢 Memory Leak Test Results13 passed, 0 leaked out of 13 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (13)
|
Avoid treating the first arbitrary body tracking frame as the default bind pose, and make first-frame capture opt-in for calibrated starts. Apply hand aim/twist correction in the direct retarget path so startup palm orientation does not bake a session roll offset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
4dd3898 to
2bef641
Compare
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
|
WebGL2 visualization test reporter: |
|
Visualization tests for WebGPU |
🟢 Memory Leak Test Results13 passed, 0 leaked out of 13 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (13)
|
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
|
WebGL2 visualization test reporter: |
|
Visualization tests for WebGPU |
🟢 Memory Leak Test Results13 passed, 0 leaked out of 13 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (13)
|
Problem
Fixes https://forum.babylonjs.com/t/quest-3-webxr-body-tracking-hand-forearm-orientation-can-initialize-incorrectly/63541/5 (post #5 — the startup roll issue).
When entering WebXR body tracking with the palm facing up or sideways, the avatar's hand would have a permanent ~180° roll offset for the entire session. The first issue (asymmetric start-time behavior) was already fixed in a previous PR; this PR addresses the follow-up startup roll bug.
Root Cause
The delta-from-bind retarget path uses an aim + twist correction to make joint orientation independent of the startup pose. The twist correction requires
_bindLocalTwistNormal— a vector representing the "palm plane" direction in the avatar's bind-rotation space.This vector is ideally derived from the avatar's bind-pose skeleton during
setBodyMesh.HandTwistReferenceJointslistedLEFT/RIGHT_HAND_LITTLE_METACARPALas the second spread point for the palm plane.findBestUnmappedDescendantForJointtokenizes that name and looks for the token"little"— but Mixamo rigs name this bone "Pinky" (e.g.mixamorig:LeftHandPinky1), scoring 0. The bind computation falls back to the XR startup positions, which vary with the user's hand orientation at session entry, baking in a permanent roll offset for the whole session.Fix
Change the second twist-plane reference from
LEFT/RIGHT_HAND_LITTLE_METACARPALtoLEFT/RIGHT_HAND_RING_METACARPAL. The token"ring"matches Mixamo'sLeftHandRing1bones (score ≥ 1), so the avatar bind-pose computation succeeds and the twist normal is startup-pose independent.Other changes
_computedDirectWorldRotationsto avoid per-frame allocations in_retargetDirect_retargetDirectfallback pathIWebXRBodyTrackingOptions.autoCaptureBindOnFirstFrameoption (defaultingtrue) for apps that want an explicit calibration-pose workflow instead of auto-captureTesting
All 84 unit tests pass. New regression test: verifies that a palm-up startup and a neutral startup produce the same avatar orientation for the same live hand pose.