Skip to content

[NV] Add LiDAR sensor support for 3D Gaussian Splatting rasterization#890

Merged
vince-brisebois merged 20 commits into
mainfrom
nv/07-lidar
Mar 14, 2026
Merged

[NV] Add LiDAR sensor support for 3D Gaussian Splatting rasterization#890
vince-brisebois merged 20 commits into
mainfrom
nv/07-lidar

Conversation

@rodlima78

Copy link
Copy Markdown
Collaborator

Summary

  • Lidar camera model: Introduces a spherical-coordinate lidar sensor with configurable elevation/azimuth beams, supporting both uniform and non-uniform beam distributions, plus rolling-shutter pose refinement.
  • Lidar tiling and intersection: Adds a dedicated tile-based intersection pipeline (IntersectTileLidar.cu) that partitions the spherical field of view into azimuth/elevation tiles and assigns Gaussians to
    tiles based on their projected 2D covariance ellipses.
  • Lidar rasterization: Extends the rendering pipeline (rendering.py, _wrapper.py) to support lidar sensors end-to-end — projection, tiling, alpha-compositing — reusing the existing 3DGS/3DGUT rasterization
    kernels with distance-based output.
  • Python reference implementation: Adds pure-PyTorch lidar equivalents (_torch_impl_lidar.py, _torch_lidars.py) for CPU testing and cross-validation against the CUDA kernels.
  • Bug fixes and robustness: Fixes singular matrix inversion in SH evaluation, corrects hit-distance clipping when global_z_order==false, adds a torch.tensor_hash fallback for PyTorch < 2.9, and removes
    premature culling of small Gaussians.
  • Comprehensive test coverage: Adds extensive tests for lidar projection, tiling, intersection, and rasterization, as well as a new 3DGUT distance-output rasterization test.

Key details

  • Lidar beam layout is defined by per-beam elevation/azimuth angles plus a beam-to-pixel mapping, allowing irregular scan patterns (e.g., Velodyne, Ouster, Hesai).
  • Tiling operates in spherical coordinates — azimuth bins form columns, elevation bins form rows — with special handling for azimuth wraparound across the 0°/360° boundary.
  • Rolling-shutter support iteratively refines Gaussian positions using per-column ego-motion transforms, controlled by rs_num_iters.
  • global_z_order=false switches rasterization from depth-based to hit-distance-based ordering, which is the natural sorting for lidar returns.
  • The TensorView::data() helper was added to simplify raw-pointer access in CUDA kernels.
  • NUM_CHANNELS must include 1 in the build to support single-channel (intensity-only) lidar rendering.

@rodlima78
rodlima78 force-pushed the nv/06-extra-signals branch from d4abcdd to 2cbc551 Compare March 14, 2026 02:37
@rodlima78
rodlima78 force-pushed the nv/06-extra-signals branch 2 times, most recently from 5fb18f9 to 2f16f71 Compare March 14, 2026 03:19
@vince-brisebois
vince-brisebois force-pushed the nv/06-extra-signals branch 2 times, most recently from 6d1148a to 026c361 Compare March 14, 2026 03:35
Base automatically changed from nv/06-extra-signals to main March 14, 2026 03:42
@vince-brisebois
vince-brisebois merged commit 6ac6e36 into main Mar 14, 2026
@vince-brisebois
vince-brisebois deleted the nv/07-lidar branch March 14, 2026 03:44
liruilong940607 added a commit that referenced this pull request Apr 30, 2026
  Fix empty API pages on docs.gsplat.studio/main

The API documentation pages (utils, rasterization, compression,
strategy) have been empty since the LiDAR PR (#890)
was merged. The pages render with section headers but no function or
class documentation.

Root cause: The docs CI pins torch==2.0.0, which is incompatible with
modern setuptools>=82 (removed pkg_resources).
When Sphinx tries to import gsplat, the import chain reaches _backend.py
which does import torch.utils.cpp_extension
at module level — this fails, and Sphinx silently skips all
autofunction/autoclass directives, deploying empty pages.

Fix: Unpin torch so the CI installs a version compatible with the
current setuptools.

  Also fixes three pre-existing Sphinx warnings in the docs:
  - Mismatched backticks in fully_fused_projection docstring
  - Unindented table body under a .. table:: directive in eval.rst
  - Duplicate here link targets in eval.rst (use anonymous refs)

Tested locally — all API pages now render correctly (utils: 16,
rasterization: 2, compression: 3, strategy: 9
  documented symbols).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants