Alibaba’s latest artificial intelligence (AI) article features VQRF, a new compression framework designed for volumetric radiation fields like DVGO and Plenoxels.

Due to its potential use in several virtual reality and augmented reality applications, the subject is gaining more and more importance. When given a collection of photos taken from various viewpoints with known camera postures, a new view synthesis seeks to achieve a photorealistic representation of a 3D scene at undiscovered perspectives. Neural Radiation Fields (NeRF) successfully modeled and rendered 3D scenes using deep neural networks. These networks are trained to map each 3D position given a viewing direction to its associated view-dependent color and volume density using volumetric rendering techniques.

Due to the reliance of the rendering process on selecting a huge number of points for sampling and passing them through a complicated network, there is a huge computational cost during training and inference. Voxel-based structures can dramatically increase the efficiency of training and inference, as evidenced by recent improvements after reconstructing radiation fields. These volumetric radiation field methods often store features and retrieve sample points (such as color features and volume densities) by efficiently trilinear interpolation without neural networks. They have a small neural network installed.

They have replaced complex networks. However, using volumetric representations still incurs significant storage costs, such as the over 100 terabytes needed to represent the scene in Figure 1, which makes it impractical for use in real-world scenarios. Voxel grids have a storage problem that must be solved while preserving rendering quality. To better understand the characteristics of the grid patterns, the distribution of voxel importance scores was estimated. Only 10% of the voxels contribute more than 99% of the importance scores of a grid model, which shows that the model has a lot of redundancy.

The method they propose to compress volumetric luminance fields allows for a 100% reduction in storage compared to the original grid models while maintaining comparable rendering quality. Figure 2 shows an illustration of the framework. The suggested framework is not architecture specific but extremely broad. The framework includes three processes: voxel slicing, vector quantization, and post-processing. Least significant voxels that dominate the model size while making minimal contribution to the final render are removed via voxel pruning. Using a cumulative score rate measure, they present an adaptive pruning threshold selection technique, making the pruning strategy applicable to various scenes or base models.

By creating importance-aware vector quantization with an efficient optimization strategy, they propose to encode important voxel features in a compact codebook to further reduce model size. A joint tuning mechanism encourages compressed models to approximate the rendering quality of the original models. Finally, they perform a fast post-processing step to obtain a model with low storage cost. As shown in Figure 1, for example, a model with a storage cost of 104 MB and a PSNR of 32.66 can be compressed into a model with a cost of 1.05 MB and minimal visual quality loss (PSNR of 32.65).

To validate the proposed compression framework, they undertake extensive experiments and practical investigations that demonstrate the effectiveness and generalizability of the proposed compression pipeline across a wide variety of different volumetric approaches and circumstances.

Check Paper and GithubGenericName. All credit for this research goes to the researchers on this project. Also don’t forget to register. our Reddit page and discord channelwhere we share the latest AI research news, cool AI projects, and more.