# To all reviewers

We thank the reviewers for their insightful comments and valuable suggestions. We are very excited that the reviewers appreciated the novelty and soundness of our approach (*i.e.,* integrating deep generative models with prior art on mapping) [**Reviewer QFY4, Reviewer SFD9**], found the paper interesting and well-written [**Reviewer QFY4, Reviewer SFD9**], and acknowledged our extensive evaluation and impressive results on the large-scale synthetic dataset [**Reviewer QFY4, Reviewer SFD9**].

## Novelty and technical contributions
As demonstrated in the paper (and further below), *simultaneous generation and mapping is the key to producing a large-scale, realistic, globally consistent 3D world*. Specifically, by grounding scene generation with mapping, one can generate a diverse set of scenes that *are coherent with* existing appearance and structure; it also allows one to reproduce mapped regions *with consistency*. Through iteratively updating the map, one can further expand the generation process to an extremely large scale *without drifting*.

We strongly believe SGAM is a critical and innovative step towards perpetual 3D scene generation.Through this paper, we also hope to convey the importance of explicitly 3D modeling in large-scale scene generation. While we indeed exploit VQ-GAN and KinectFusion in SGAM (*i.e.*, leverage KinectFusion for volumetric map building, adopt VQ-GAN for generative sensing, etc), *why they are used* and *how they are used* are all carefully designed. The resulting framework is generic, interpretable, and can be applied to various setup. It is not just a simple extension. Also, exploiting existing algorithms to realize a novel idea does not mean there is no technical contribution. We hope the reviewers, in particular **Reviewer GmLE**, can acknowledge this.

We now address the concerns of each reviewer individually. We have also included new experimental results per reviewers' request in the revision (highlighted in blue). We strongly encourage the reviewers to take a look.

---

# Responses to Reviewer QFY4

**Robustness to depth measurement:** Our approach *does not* require accurate depth, even during training. In fact, the "GT depth" of our GoogleEarth-Infinite dataset contains noise. The "GT depth" is obtained by rasterizing the coarse meshes (which are built from SfM and MVS point clouds) that we crawled with the Google API. Therefore by nature it is merely a proxy geometry of the real world and is far from accurate. Fortunately, with the help of the VQ-based generative sensing module, we can learn to de-noise with the quantized codebooks and produce perpetual 3D scene without drifting. We also note that we use the same "GT depth" to train all baselines as well as our method. To further showcase the robustness of our approach, we are conducting experiments on KITTI-360. Specifically, we use the stereo estimation from deep nets to serve as the "GT" to train our model. Due to computational constraints, the training is still in progress. We, however, still provide some promising preliminary results in the revision for the one-step prediction results. Please see the response to Reviewer SFD9 for more details on KITTI.

**Results on real world dataset:**

We stress that our goal is to enable large-scale, long-term, globally consistent, perpetual 3D scene generation. Our method can generate at a much larger scale without domain drift. To verify our claim, we compute the FID score across all competing on GoogleEarth-Infinite by unrolling the perpetual generation for 60 steps with different initial images. All the methods follow the same predefined trajectory. Results show that our method significantly outperforms other methods:

FID scores on generated GoogleEarth-Infinite dataset (unrolling 60 frames)
| InfiniteNature | GFVS-implicit | GFVS-explicit |Ours |
| -------- | -------- | -------- |-------- |
| 182.6 | 160.4 | 133.1 | 79.26 |

We also would like to highlight the qualitatitive results shown in Fig. 7.

We also note that InfiniteNature predicts the next frame by directly warping pixels and refining the results. In the short term, such an inpainting-like scheme looks slightly faithful (since it aims to fill the residual) compared to the images generated from the discrete bottleneck, resulting in superior one-step prediction performance. However, we find the artifacts will thus accumulate over time and results in severe drifting compared to GFVS and ours. In contrast, ours uses a vector quantization in latent space, inducing strong prior and constraints. This expressiveness vs. prior trade-off makes our approach slightly worse than Infinitenature in one-step prediction (PSNR: 23 vs. 24, higher is better). Nevertheless, this discrete bottleneck helps prevent SGAM from domain drift, resulting in significantly better results long-term image generation quality (FID: 79 vs. 182, lower is better).

**Flickering in videos:** Thanks for pointing this out! It is simply a visualization bug when we generate the video. The actual generation is continuous and the produced scene changes smoothly and coherently.

---

# Responses to Reviewer SFD9

**Limitation on voxel scene representations:** We agree that the resolution of the voxels will impact how much fine-grained details we can capture. It is a hyper-parameter that needs to be determined. The voxel-hashing we used for mapping is a popular strategy for scaling up volumetric representation at high precision. One can potentially adopt other special data structures such as Octotree to reduce memory usage and computational cost, besides voxel hashing. Thanks for the suggestions! We will include relevant discussion in the final version.

**Results on complex scenes:** To validate whether our method can handle more challenging scenarios and first-person perspectives, we train SGAM on KITTI-360 dataset. KITTI-360 is a challenging self-driving dataset. It contains rigid objects (*e.g.,* cars), fine-grained geometry (*e.g., trees), reflections, and most importantly noisy sensory data. This allows us to evaluate how robust SGAM is. Our model is currently still training.
Based on the the intermediate checkpoint, we can achieve **PSNR 23.80, SSIM 0.838, LPIPS 0.122, FID 37.88** on one-step prediction. Some preliminary qualitative results are shown in the revision. Due to computational resource limits, we will include the results on indoor synthetic scenes in the final version. We thank R2 for the great suggestion. We choose KITTI-360 since it allows us to verify multiple aspects at the same time (*e.g.,* robustness to the depth measurement).

**On the convergence of training:** As mentioned in Sec. 3.4 and Sec. 4.3, we adopt a two-stage training strategy. Each stage is crucial. Missing either stage could have a detrimental effect. The color biases in CLVER arises from the environmental map that we used, rather than the random seed. We will change the environmental map and re-train our model to mitigate the issue.

**Generating scenes from less structured trajectories:** The scenes shown in the paper are indeed generated with fixed camera angles. To demonstrate that SGAM is able to generate scenes from less structured trajectories and different camera viewpoints, we exploit two new setup: (i) we adopt a spiral trajectory and rotate the cameras along the yaw axis such that they always "look at" the center of the scene; (ii) we fixed the origin of the camera and rotate 360 degrees along its pitch axis. Due to computational resource limits, we did not re-train our models with various camera angles. Instead, we simply adopt the model trained with fixed angles and perform scene generation. Surprisingly, despite such a domain gap, SGAM is still able to produce coherent, reasonable 3D scenes. We refer the readers to the revision for the qualitative results (see supp **Figure 2** ).

---

# Responses to Reviewer GmLE

**Simultaneous generation and mapping doesn't make sense.**

We strongly believe SGAM is a critical and innovative step towards perpetual 3D scene generation.Through this paper, we also hope to convey the importance of explicitly 3D modeling in large-scale scene generation. While we indeed exploit VQ-GAN and KinectFusion in SGAM (*i.e.*, leverage KinectFusion for volumetric map building, adopt VQ-GAN for generative sensing, etc), *why they are used* and *how they are used* are all carefully designed. The resulting framework is generic, interpretable, and can be applied to various setup. It is not just a simple extension. Also, exploiting existing algorithms to realize a novel idea does not mean there is no technical contribution. We hope the reviewers, in particular **Reviewer GmLE**, can acknowledge this.

**Repeatedly applying VQ-GANs will not work.**

We want to highlight our task is a full 3D generation instead of perpetual 2D image generation on the image plane. We are **NOT** repeatedly applying VQ-GAN, which will not work for this task.
We need at least two critical components to achieve our goal of simultaneously perpetual sensor and 3D shape generation: 1) render novel views at a new pose by reflecting the stereo-parallax. 2) building a 3D representation ensure consistency over the long run. Solving these two challenges is the core intellectual merit of this paper.

**Experiments on real-world data:**

We want to highlight that GoogleEarth-Infinite is a challenging real-world dataset that we collect from GoogleEarth. The FID scores shown below demonstrate our performance for long-term video sequence generation.

FID scores on generated GoogleEarth-Infinite dataset (unrolling 60 frames)
| InfiniteNature | GFVS-implicit | GFVS-explicit |Ours |
| -------- | -------- | -------- |-------- |
| 182.6 | 160.4 | 133.1 | 79.26 |

**Camera poses beyond near nadir views**

The scenes shown in the paper are indeed generated with near nadir views. To demonstrate that SGAM is able to generate scenes from less structured trajectories and different camera viewpoints, we exploit two new setup: (i) we adopt a spiral trajectory and rotate the cameras along the yaw axis such that they always "look at" the center of the scene; (ii) we fixed the origin of the camera and rotate 360 degrees along its roll axis. Due to computational resource limits, we did not re-train our models with various camera angles. Instead, we simply adopt the model trained with fixed angles and perform scene generation. Surprisingly, despite such a domain gap, SGAM is still able to produce coherent, reasonable 3D scenes. We refer the readers to the revision for the qualitative results (see supp **Figure 2** ).

In addition, following the reviewers suggestion, we are also running SGAM on the KITTI-360 dataset, an urban self-driving dataset from a first-person perspective. Due to time and resource limits, the model is currently training. Validation performance shows promising results in the generative sensing model.

---

# Letter to AC.

Try quote violating reviewer guideline.