# HRFuser Rebuttal Answers

## General comments

**Character count: 4986 - OK**.

We would like to thank all three Reviewers sincerely for their constructive remarks on our paper and for acknowledging the generality and versatility of the proposed sensor fusion architecture, the novelty of our fusion block, the high-quality experimental results on multiple autonomous driving datasets and in adverse conditions, the thoroughness of the conducted ablation studies, and the clarity in the presentation of the material.

We structure this rebuttal as follows:
* In this comment, we address the concerns that were shared by more than one Reviewers.
* Concerns that were specific to a single Reviewer are addressed in the respective response to that Reviewer.

**Justification of addressing 2D detection versus 3D detection.**
Robust performance in all weather conditions is fundamental for level-5 autonomous cars. We propose a multi-modal architecture that aims to combine complementary sensors to yield such robust performance. 2D detection is better suited than 3D detection for annotating adverse conditions, as especially lidar is severely affected by adverse weather and objects tend to include very few (if any) points for 3D annotation. As highlighted in the supplemental material of STF [2], missing 3D bounding boxes are only annotated in 2D. Using 2D annotations thereby allows us to identify objects even when the lidar is impaired.

As detailed in our paper (L35--44), most fusion works for visual perception in autonomous cars focus on 3D detection, in which the rich textures from cameras provide additional information besides the 3D signal from the primary sensor (lidar). However, performing recognition in the 2D image space is also fully relevant for autonomous cars; a good piece of evidence for this are the several 2D tasks on which extensive research has been performed with various autonomous driving datasets in recent years, e.g. semantic/instance/panoptic segmentation as well as 2D object detection. The 2D recognition outputs from models that perform such tasks can be used by downstream modules in autonomous vehicles, without the need to provide these downstream modules with 3D outputs, as evidenced in [A], where 2D semantic segmentation is found to be among the two most useful vision-level cues for executing actions.

For the task of 2D object detection, we have mentioned (L35--44) that very strong camera-based models have been presented. In this context, our contribution is *orthogonal* to sensor fusion for 3D detection, as we show that leveraging additional modalities on top of the primary camera modality can substantially improve performance for 2D detection even when starting from the aforementioned strong camera-based baselines, as opposed to adding information from the camera on top of lidar, which is done in 3D detection.

Besides, our work is in no way an outlier with regard to focusing on 2D detection in multi-sensor fusion. As we mention in the paper (L122--125), there are multiple works that address 2D detection with radar and camera [7, 13, 46, 48, 65, 84] or lidar and camera [1, 3, 15, 17, 44]. In fact, for papers that focus on radar and camera fusion, it is more common to consider 2D rather than 3D detection; we have found only one work [47] pertaining to the latter case in our literature review. Moreover, the very few works [2, 52] that consider three or more modalities (camera, lidar, radar, gated camera) also focus on 2D detection, as we do.

**Inference speed and number of parameters/flops with fusion architecture.**
We have followed the Reviewers' request and provide an extension of Table 4 in the following table, showing that our fusion method adds only a minor computational overhead. Even when using all three additional modalities besides the camera, the flops are increased by less than 30% and the parameter count by a marginal 2%. The inference time of the full multi-modal network is much less than double the time of the camera-only network. We will include this comparison in the supplement if the paper is accepted.

C: RGB camera, R: radar, L: lidar, G: gated camera
|Modalities|Flops (GFLOPs)|Parameters (M)|Inference Time (ms)|
|-|-|-|-|
|C|104.0|47.9|91.6|
|CL|114.1|48.8|119.8|
|CLR|123.3|49.4|142.2|
|CLG|123.2|49.4|140.9|
|CRG|123.2|49.4|138.2|
|CLRG|132.4|49.9|159.5|

**Specifications on nuScenes classes used in experiments and class-wise performance.**
As we mention in the paper (L233--235), ''unless otherwise stated'' we indeed use the original set of 10 nuScenes classes in our experiments. The only case where we use a reduced set of 6 classes is in Table 1, in which this choice of classes was necessary for comparability to the method in [46], which only reports results on these 6 classes. This different choice of classes has been explicitly stated in the caption of Table 1. We will also explicitly state the aforementioned reason for this choice if the paper is accepted.

[A] B. Zhou, P. Krähenbühl, and V. Koltun. Does computer vision matter for action? Science Robotics, 4(30), 2019.

## Reviewer zy9g Answer

**Character count: 4683 - OK**

We thank Reviewer zy9g sincerely for their valuable feedback.

**Analysis of what our MWCA fusion block learns.**
In order to provide more insights into what MWCA learns, we included some attention maps for the cross-attention in section 8 (L73--85) of the supplemental material. MWCA attends to each modality specifically, reflecting each sensor's characteristics. It learns to reason continuously across windows, as demonstrated by the continuous nature of the attention maps. This is especially noticeable in the attention map of the gated camera, where the highlighted edges are continuous and uninterrupted by the boundaries of the local windows.

**Overhead on inference speed and number of parameters with fusion architecture.**
We point the Reviewer to our general comment for a detailed response to this point.

**Effect of selection of primary modality on performance.**
In the paper, we have selected the camera as the primary modality for our HRFuser architecture. Following the Reviewer's request, we experiment on the STF dataset by setting each available sensor besides the camera, i.e., lidar, radar, and gated camera, as the primary sensor, and provide a comparison on the various test sets of STF in the following table. We observe that having either the camera or the gated camera as the primary sensor generally attains higher performance than having lidar and radar as the primary sensor, even though the respective difference is slight. Our intuition for this finding is that the high spatial resolution of the camera and the gated camera makes them better choices for serving as the primary modality, as the primary modality is used in our cross-attention block to compute the queries and thus to determine which regions of the other modalities to attend to, a function which can be carried out with higher spatial accuracy in case high-resolution readings from the primary modality are available. However, since fusion across all modalities starts at an early stage in the network, HRFuser can learn meaningful features even when using a sparse modality such as radar as the primary modality, and in any case, HRFuser is fairly robust with respect to which modality serves as the primary one.

|Primary Sensor|clear|||light fog|||dense fog|||snow/rain|||
|-|-:|-|-|-:|-|-|-:|-|-|-:|-|-|
||easy|mod.|hard|easy|mod.|hard|easy|mod.|hard|easy|mod.|hard|
|RGB camera (ours)| 90.15|**87.10**|79.48|90.60|89.34|**86.50**|87.93|80.27|78.21|**90.05**|**85.35**|**78.09**|
|Lidar|90.02|86.89|79.35|90.61|**89.45**|86.28|88.64|80.60|78.59|89.86|84.98|77.44|
|Radar|89.99|86.96|79.47|**90.65**|89.30|80.89|88.45|80.53|72.33|89.86|85.08|77.46|
|Gated camera|**90.20**|86.82|**79.54**|90.58|89.33|80.92|**88.73**|**80.87**|**79.05**|90.03|85.29|77.82|

**Justification of addressing 2D detection versus 3D detection.**
We point the Reviewer to our general comment for a detailed response on this issue.

**Specifications on nuScenes classes used in experiments.**
We point the Reviewer to our general comment for a detailed response on this issue.

**Class-wise performance on nuScenes.**
We also report below the average precision of our HRFuser-T versus the baseline camera-only HRFormer-T on each individual class on nuScenes. HRFuser consistently outperforms HRFormer across all classes by large margins. This class-wise comparison will be added to the supplement if the paper is accepted.

|Class|HRFormer-T (AP)|HRFuser-T (AP)|
|-|-|-|
|car|50.2|**53.1**|
|truck|29.2|**36.8**|
|trailer|14.9|**20.6**|
|bus|39.4|**48.1**|
|construction vehicle|7.0|**9.6**|
|bicycle|20.8|**26.6**|
|motorcycle|22.3|**28.8**|
|pedestrian|25.4|**30.7**|
|traffic cone|26.8|**28.6**|
|barrier|28.8|**32.0**|

**Overview of complementary features of different sensors.**
Due to the space limitations for the paper, we have only briefly outlined the complementary features of different sensors we consider in the Introduction (L27--29) to motivate the need for sensor fusion. Following the Reviewer's request, we provide here a more detailed overview of these features for each sensor:
* Camera: Very high resolution and rich texture, but poor readings in low illumination and fog and no direct geometric information.
* Lidar: Fair resolution, explicit range information and independent from external illumination, but degradation when the optical medium is not clear (fog, rain, snow).
* Radar: Robust to adverse weather and illumination, velocity information, but low resolution and noisy.
* Gated camera: High resolution, robust to adverse weather and illumination, but still not widely adopted.

We will extend the comparison of the sensors in the introduction to include the most important of the above points if the paper is accepted.

## Reviewer opAV Answer

We thank Reviewer opAV sincerely for their valuable feedback.

**Justification of addressing 2D detection versus 3D detection.**
We point the Reviewer to our general comment for a detailed response on this issue.

**Choice of datasets for experiments.**
To train and test HRFuser with focus on robustness to weather conditions, we selected nuScenes and STF. These datasets offer a diverse set of sensors and conditions, while KITTI only includes camera and lidar data in relatively favorable conditions.

**Need to render additional modalities in the camera frame.**
Rendering modalities in the image space does not constitute a major limitation for practical automotive applications, as the intrinsic and extrinsic parameters of each sensor are usually known. Focusing on 3D detection instead would also not lift the need for calibration, since projecting the camera into 3D space still requires accurate calibration.

**Specifications for nuScenes classes used in experiments.**
We point the Reviewer to our general comment for a detailed response on this issue.

**Justification of using HRFuser-T on STF.**
Compared to nuScenes, STF consists of a relatively small training dataset, since most frames contain adverse-condition data that are not used for training. To avoid overfitting and to conserve resources, we choose to report the results for HRFuser-T.

**Effect of a shortcut connection for $Y_p^\beta$.**
We point the Reviewer to our supplemental material (L44--49) and Table 9 therein, where we compare HRFuser with and without such a shortcut connection. When removing the skip connection, we observe a drop of 0.6% in AP relative to our default MWCA. This finding indicates that skip connections from all modalities are beneficial for cross-attention, as it allows the network to attend to details without having to learn the identity function.

**Missing shortcut for alpha in Figure 3.**
We point the Reviewer to the right side of Figure 3, where the black line indeed depicts the shortcut for alpha.

## Reviewer hDjJ Answer

**Character count: 4981 - OK**

We thank Reviewer hDjJ sincerely for their valuable feedback.

**Novel contributions.**
The paper presents a novel, versatile and scalable architecture for sensor fusion:
* We propose a novel multi-window cross-attention (MWCA) block to simultaneously fuse features from multiple modalities while addressing the quadratic complexity of attention.
* We present HRFuser, a multi-modal fusion architecture. As highlighted by Reviewer opAV, the structure of HRFuser represents novelty. Adapting the basic camera-only network to handle multiple modalities in a scalable manner involves non-trivial design choices.
* We ensure scalability w.r.t. the number of sensors, by fusing modalities in a sensor-agnostic fashion and restricting secondary modalities to one resolution, which minimizes the computational overhead of fusion.

We agree that neither the idea of multi-modal embeddings nor the finding that radar or gated camera work better in adverse conditions is new, but we do not claim these as our novelties. Our claim lies rather in the novelties mentioned above.

**Absence of theoretical results.**
This paper lies in the area of computer vision (CV), which is solicited in the NeurIPS call for papers. In CV, it is common practice for papers to be primarily based on empirical findings, which are backed by the researchers' intuition based on their domain-specific knowledge. While our paper indeed does not present any theoretical results, the mathematical formulation of our method, in particular our multi-window cross-attention block, is sound and rigorous.

**Justification of addressing 2D detection versus 3D detection.**
We point the Reviewer to our general comment for a detailed response on this issue.

**Concerns about evaluation on nuScenes.**
COCO is a standard metric for 2D detection, but the choice of metric does not have an impact on the general trend of the results, as shown by our improvements on STF, which are reported with the KITTI evaluation metric.
We acknowledge (L236--237) that for 2D detection evaluation on nuScenes we are restricted to the validation set. However, our state-of-the-art results on STF show that our results on nuScenes are not due to overfitting, but rather that our approach works well across multiple settings.

**Generalization of the multi-modal representations to different domains.**
For intra-dataset shift due to changes in weather conditions, we have shown in Table 4 that HRFuser achieves a substantially improved generalization to unseen conditions.
For inter-dataset shifts, we cannot expect either the uni-modal or the multi-modal representations to generalize well, as different datasets are recorded with different sensors. This limitation is ubiquitous in CV methods and it is an area of active research (domain adaptation), but to address it is beyond the scope of our work.

**Ablation of radar.**
We thank the Reviewer for pointing out this missing case (RGB+radar) from Table 4. We address their request in the following table. The RGB+radar network substantially outperforms the RGB-only network. We have already shown the utility of adding the radar on nuScenes in Table 5.

|Modalities||clear|||light fog|||dense fog|||snow/rain||
|-|-|-|-|-|-|-|-|-|-|-|-|-|
||easy|mod.|hard|easy|mod.|hard|easy|mod.|hard|easy|mod.|hard|
|RGB|79.81|62.48|53.68|80.84|63.07|62.08|71.84|62.69|54.05|78.68|61.19|52.72|
|RGB + radar|**88.48**|**80.15**|**76.25**|**90.37**|**86.40**|**79.60**|**88.51**|**79.72**|**71.87**|**88.13**|**78.85**|**70.27**|

**Usage of more data by models with more modalities.**
To address the Reviewer's concern, we trained HRFuser-T on a subset of nuScenes containing half the training data. As seen in the table below, this model substantially outperforms the camera-only model trained on the complete training set, recovering ~90% of the performance gain of the fusion model that sees the complete training set. Thus, even with a smaller volume of data, HRFuser delivers a significant performance improvement.

|nuScenes (%)|Modalities|AP|
|-|-|-|
|100|C|26.5|
|50|CL|30.8|
|100|CL|31.2|

The volume of information is not the same across modalities. In nuScenes, only 0.71% of the pixels per radar image and 1.61% per lidar image have a measurement on average. Thus, adding a second modality does not double the volume of information but only increases it slightly, which means that the comparison of a camera-only model to a fusion model is reasonably fair.

We argue that utilizing all training data from multiple modalities is not a weakness but rather a strength of HRFuser. Additional modalities are typically available in automotive datasets and, as mentioned in the paper (L264--266), their usage helps avoid overfitting of large models.

**Inference speed and number of flops.**
We point the Reviewer to our general comment for a detailed response to this point.

**Inclusion of limitations.**
We mention limitations in the paper (L309--310 & L323--324) by showing failure cases (Figures 4 & 5 + Figures 9 & 11 of the supplement).

## Area Chair Answer

We honestly believe that the rating which Reviewer hDjJ has provided for the Soundness of our paper (1 poor) is unjustified. More specifically, the mathematical formulation of our method, in particular our multi-window cross-attention block, is sound and rigorous, and Reviewer hDjJ has not raised any concerns about it. We hypothesize that their reasoning for this rating lies in the fact that our paper "does not have any sound theoretical finding", which is true but does not have any relation with the soundness of the presented theoretical formulation of our method. Soundness refers to the correctness of included material, not to the absence of additional material.