3D Gaussian Splatting for Real-Time Radiance Field Rendering
Ғылым және технология
SIGGRAPH 2023
(ACM Transactions on Graphics)
----------------------------------------------------
repo-sam.inria.fr/fungraph/3d...
Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates.
We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (≥ 100 fps) novel-view synthesis at 1080p resolution.
First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.
Пікірлер: 58
What a time to be alive!
Now this is epic.
Very simplistic explanation.
Oh my God. I'm shaking right now
Wtf this is like pure magic. Amazing work, can’t wait to see it in action!!
This is insane. Good work.
can we say that is just amazing !!
Really impressive, congrats.
That's amazing! Good job : O
The future of photography 😱
Wow Awesome work looks stunning! I would love to do a Vulkan implementation.
C'est magnifique
Mind blowing
amazing
What an incredible achievement, how do I get my hands on this? :o
The future will be representation. I wonder if there is enough viewing angles on smartphones for real time representation.. or maybe a sort of setup with multiple cameras? Or maybe the LiDAR on an iPhone has enough depth for representation ?
I don't understand any of the technical terms but that looks like dark magic to me. This is like, good enough to use in high end films now.
every damn day we get closer to a hype realistic optimized vr game
Amazing
The future of gaming just got set.
Waiting for this rendering tech to be available in Blender, Unreal and Unity 😊
@trollenz
7 ай бұрын
I've seen plugins for unreal and unity
Every day we get closer to the holodeck
This is amazing! Since it's able to render orders of magnitude faster, is it also able to capture moving objects in the camera view?
@dan_obie
9 ай бұрын
I'm interested in this as well. I imagine that the moving object would ultimately be considered noise and be culled. What happens to leaves and tree branches as they move during image capture?
Waiting for a tutorial for how to setup 😍
Just imagine the future: an instant 3D models mrom a mobile phone camera!
How are non-lambertian (i.e. viewpoint dependent) effects, like fresnel, specular, etc. achieved with this approach? It appears as though the technique struggles to represent glass (e.g. the reflections and transparency of the windscreen as shown in the project page). I suppose one approach may be to optimize spherical harmonics coefficients which are later pruned after training, though this may result in really bad ringing. Or you could have a "cone of influence" (cone orientation quaternion, anisotropic cone fov) that defines the opacity of a given gaussian depending on the viewing angle. Unfortunately this would likely massively increase the number of gaussians you'd need for certain surfaces, and you may need to know how many to allocate for these surfaces ahead of time for initialization. Third, you could have a very sparse voxel grid of learned functions that converts the viewing angle from the gaussian center position, and the gaussian center position within the voxel cell, to rgba. This may be the fastest, as you'd only need to evaluate this once per gaussian per frame, not per pixel.
@arsenurano4164
10 ай бұрын
It's written in the paper, sh coefficients is optimized during training too
Awesome! I want to see those scenes in VR! 🥽
@c016smith52
9 ай бұрын
Me too, I hope this is something that could port into something on Quest, via Unity/VR or otherwise!
finally something that is easy to interpret and to work with. no idea how come they are able tor render novel views that fast. if every scene has >100k splats, I would pre-render them into bilboard sprites, but how come reflections are so good?
@LeePenkman
9 ай бұрын
The colours are stored with some extra view dependant properties which I think helps
@naninano8813
8 ай бұрын
@@LeePenkman I since found out that one of splat's parameters (also learned) are spherical harmonic coefficients. and there is a marvelous breakdown of all the math in "Spherical Harmonic Lighting: The Gritty Details". easy read. still reading it.
@adhirajdeshmukh6813
6 ай бұрын
@@naninano8813 Thanks for referring this.
1. Wouldn't it be more accurate to compare 3dgs to nerf when both are either initialized at random or both with sfm points? 2. Since this method is based on rendering and not ray tracing, how can it do such a good job at modeling the sun reflecting of a shiny object like the countertop only at certain angles?
wow
🤯
Will this be available for public/professional use soon?
any comparisons with mobilenerf?
can we generate the point cloud instead of capture it from real life? i'm thinking of game development use cases
Cyberpunk 2077 brain dance irl
I wonder if you could say train an ai on Gaussian splatters of animals that come from the input of dna sequences if you could get it to output mammoths
@Universaa
9 ай бұрын
it will certainly be intriguing.
Streetview is about to become the biggest game map ever
what gets optimized during backprop?
@elfferich1212
9 ай бұрын
gaussians
@andybrice2711
9 ай бұрын
Essentially: A cloud of blurry ellipses. Their dimensions and rotations, and what colour they appear from different viewing angles.
@ElQaheryProductions
9 ай бұрын
@@andybrice2711 thanks!
in the 90s
Problem: neither PSNR nor plain SSIM, while very easy to compute, are particular good measures for subjective image similarities - someone better run these tests with more meaningful metrics like MS-SSIM & VMAF (which on top of being more close to human perception of visual differences are also used in many existing image quality, restoration & compression tests).
Can anyone say unlimited detail?
Deckard: Enhance...
What's a gaussian? 😐
@andybrice2711
9 ай бұрын
In this context: Basically a blurry egg-shape floating in space.
@drdca8263
8 ай бұрын
A Gaussian is the bell curve shape (or a 2D or 3D or etc. version of this)
What a time to be alive!