Ongoing project · Real-time Perception Junho Lee

Real-time Normal Gaussian Splats

A Gaussian-splat version of NFL built for instances that move. Gaussian splats update far faster than a voxel grid, which matters once the scene itself is changing in real time.

We first extract SAM2 temporal masks to track object identity across frames, then drive the splat update with a render-and-compare pipeline against the live RGB stream.

Representation

Gaussian splatting of NFL.

RGB input stream used to drive the real-time normal Gaussian splat

RGB input

Normal field reconstructed in real time as Gaussian splats

Normal field

Depth derived from the real-time normal Gaussian splat representation

Depth

Scene change

Dynamic scene.

A scene where objects move and the normal splat representation updates live

Scene where objects move

SAM2 temporal masks used to track object identity across frames

SAM2 masks

Per-instance results

Tracking results.

Tracked normal splat reconstruction for object 1 across the sequence

Object 1

Tracked normal splat reconstruction for object 2 across the sequence

Object 2

Tracked normal splat reconstruction for object 3 across the sequence

Object 3

Tracked normal splat reconstruction for object 4 across the sequence

Object 4