# CoRL'22 RobustTraj Rebuttal

## Responses to Reviewer N91T

### Q1: How does the adversary plan to drive the vehicle in the real-world setting?
Thanks for pointing it out. In our paper, we did include a realistic attack, named *search* proposed by Zhang et al.[1], that considers kinematical feasibility (which enables the attacker to drive along the adversarial trajectories in the real world). The evaluation results for *search* (Table B in the Appendix) demonstrated that our method is also robust to this attack.

To further address the concerns of conducting the attack in the real-world setting, we conduct additional experiments to evaluate the robustness of our method against the practical attack, namely *search-online* provided by Zhang et al. [1]. Specifically, since the adversarial does not have ground truth of other agents, they consider a more realistic attack where adversarial has a trajectory prediction model to predict the trajectory of all agents. They assume they could collect the trajectories in the first 2 seconds and then predict the future trajectories of all other agents for the next 2-4 seconds. They use this information to generate the adversarial trajectory for the next 2-4 seconds and drive along this adversarial trajectory. As a result, this attack is no longer just perturbing the existing trajectories from a dataset but deployable in the real-world. We evaluate this algorithm on both the model trained on clean data and our robust model (i.e., *RobustTraj*). The results are shown in the following. We could find that our method could still show a significant robustness improvement compared to the clean model.

> Evaluation results of the proposed methods with the *search-online* attack proposed by
Zhang et al. [1].

| Model | Robust ADE | Robust FDE |
|:----------:|:----------:|:----------:|
| epsilon | 0.5 | 0.5 |
| Clean | 6.20 | 11.95 |
| *RobustTraj* | 3.46 | 6.46 |

### Q2: Is there a training set and a test set? Or is the adversarial training on the same dataset as it's evaluated on?
Yes, we divide the dataset into a training set and a test set following the standard division defined by the nuScenes prediction challenge. We use the training data to train a robust model and use the test data to evaluate the model's robustness.

### Q3: Are the adversarial attacks ensured to be physically plausible? i.e. Are the adversarial perturbations restricted so that a car could physically follow the perturbed trajectory?
Great point! During the evaluation, we also use the adversarial attack proposed by [1] which constrains the magnitude of the adversarial perturbation and is ensured to be dynamically feasible (i.e. physically plausible). The results are shown in Table B of the Appendix. In this way, we can measure the performance of our proposed defense strategy under realistic attacks that a car could physically follow in the real world. As an example, we include some visualizations of the generated adversarial trajectories: [scenario 1](https://i.imgur.com/b4CQ8xT.gif), [scenario 2](https://i.imgur.com/EKWFGGL.gif).

### Q4: Is the adversary only allowed to change the trajectory of a single vehicle in the scene or any/all vehicles?
We assume that all vehicles could be changed since it is the strongest setting to defend against.

### Q5: Clarifications for the equations.
In equations 1 and 2, the last term encourages diversity (line 111-112 in the main paper). By such a design, the loss is low as long as one sample is similar to the ground truth future trajectories and the loss of the other samples will be ignored so that it could be as arbitrary as possible.
In equation 2, sorry for it. We ignore the optimized parameter here for simplification. We minimize the loss to update the parameters of the generator and maximize the loss to update the parameters for discriminator.
In equation 6, the total loss represents the loss for training the trajectory prediction models. Thus the adversarial loss is attained when the input is adversarial examples X+δ while clean loss is attained when the input is clean data X.

### Q6: Minor typos/unclear wordings
Thanks for pointing out these typos/unclear wordings. We have corrected them and thoroughly proofread the revised manuscript.

#### References
[1] Zhang, Qingzhao, et al. "On adversarial robustness of trajectory prediction for autonomous vehicles." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

## Responses to Reviewer zQGj

### Q1: The adversarial attack setting is not practical. Specifically, this paper studies the L_infty attack (0.5 or 1 m deviations is the maximum change for a standard car without shifting to another lane), this limit the actor behavior a lot and will not find interesting scenarios (lane change, more advanced interactions between agents, etc) that can happen in the real world.

Thanks for your question. As mentioned (lines 25-26 in the main paper), we leverage the state-of-the-art trajectory-space adversarial attack proposed by [7]. The method adds bounded perturbation on the given trajectory from the collected dataset to achieve realistic and physically-plausible attacks on trajectory histories. Although the attack is bounded by the L_infty norm, this attack could lead to severe consequences (e.g., collision) [7].

We would also like to emphasize that to the best of our knowledge, generating ***realistic and physically plausible*** adversarial scenarios is still an open research problem. As pointed out by the reviewer, [2] considers generating adversarial scenarios. However, adversarial scenarios generated in [2] are not realistic and appear to be dangerous and violate traffic rules (e.g., driving in the reverse lane as shown in the demo of the first row on their [website](https://nv-tlabs.github.io/STRIVE/)). This is because the realism of traffic scenarios is difficult to be formulated due to the complexity of traffic rules. Thus, it violates our realistic and physically plausible attack criteria.

Moreover, although the attacks to the trajectory history space are bounded, the consequences of these attacks can be significant (e.g., collision and driving off-road, according to [7]). The inconspicuous nature of these attacks also made them difficult to detect. Thus we disagree with the reviewer’s comment that the attacks are not interesting. We believe the attack we selected is a reasonable proxy to measure the worst-case performance of the model.

Finally, we conducted an additional experiment to evaluate the robustness of our method against adversarial scenarios generated by [2]. The results are shown in the following table. We demonstrated that our ***RobustTraj*** achieves 27% (0.92/3.37) better performance (on ADE) compared to models trained on clean data (***clean***).

> Evaluation results of the proposed methods with the adversarial scenarios generated by [2].

| Model | ADE | FDE | MR | ORR |
|------------|------|------|------|------|
| Clean | 3.37 | 9.52 | 0.45 | 0.17 |
| *RobustTraj* | 2.45 | 6.00 | 0.26 | 0.11 |

### Q2: Moreover there are no constraints on the vehicle trajectories that are commonly used in other papers [1-3] (e.g., kinematic bicycle models, a drivable area in the map, constraints on acceleration, velocity, etc).
In the evaluation, we indeed consider practical attacks proposed by [7] where constraints such as drivable regions and kinematic bicycle models with acceleration and velocity constraints are imposed.

### Q3: This paper assumes an overly simplified setting.
The primary goal of this paper is to study and improve the robustness of trajectory prediction models. To obtain salient and unambiguous insights, it is necessary to minimize the conflating factors in our analysis. For example, it would be difficult to analyze whether the error is from the perception model or from the trajectory prediction model if we consider a full-fledged stack with interdependent components. We also want to stress that a key finding of this paper is that even with perfect perception, a competent trajectory prediction model can be easily fooled by adversarial attacks. Hence we believe it’s crucial to diagnose the vulnerabilities of an AV stack in silos before advancing to the entire system. That being said, we will add an additional discussion regarding the scope of the study in the next revision of the paper.

### Q4: Limited technical novelty.
First of all, we would like to emphasize that the goal of our work is to build robust trajectory prediction models. Since most trajectory prediction methods employ probabilistic generative models to cope with the uncertainty in motion forecasting, it brings unique challenges for building robust trajectory prediction. To the best of our knowledge, we are the first to propose an effective solution that builds robust trajectory prediction models through adversarial training. While we agree some aspects of our method may echo that of prior research in other topics, they serve completely different purposes and th are critical to the effectiveness of our final solution. For example, the key innovation in [4] is the auxiliary batch norm design which enables the model to capture the distribution of adversarial examples and clean data. This method is less applicable in SOTA transformer-based trajectory prediction architectures (e.g., the AgentFormer we used) since they do not have the batch norm design. In [5], the authors propose to use the adversarial vertex mixup to address the adversarial feature overfitting problem, while we propose to mix benign data and adversarial examples to stabilize the changing distribution. Similarly, prior work [8] proposes to overcome obfuscation at the evaluation stage to generate strong adaptive attacks. In this work, we encounter the obfuscation gradients on a stochastic model during the adversarial training process. Prior work [6] still focuses on attacking cooperative multi-agent ***perception system*** where communication is done via sharing intermediate representations of the perception model while we hope to improve the robustness of the ***trajectory prediction model***.

To further demonstrate the insufficiency of naively mixing benign data and adversarial examples, we conduct an additional experiment by adapting [5] for our setting. The results are shown in the following table. We could observe that *AV mixup* [5] still performs worse compared to our proposed *RobustTraj*.

> Evaluation results of adversarial training with the existing methods *AV mixup* [5].

| Model | ADE | Robust ADE | FDE | Robust FDE |
|:------------:|:----:|:----------:|:----:|:----------:|
| epsilon | 0.5 | 0.5 | 0.5 | 0.5 |
| Clean | 2.05 | 6.86 | 4.41 | 13.08 |
| *AV mixup* [5] | 2.29 | 5.34 | 4.97 | 9.62 |
| *RobustTraj* | 2.14 | 3.69 | 4.36 | 7.10 |

### Q5: Evaluation on PGD attack only.
Here we include the results of the applicable white-box *AutoAttack* (i.e., APGD-CE) since APGD-DLR and FAB are designed for classification. We could observe our method (*RobustTraj*) could still achieve non-trivial robustness improvements compared to the clean model. We will add the black-box attack in the revised version given the limited time for the rebuttal.

> Evaluation results of the proposed methods with the APGD-CE attack [8]

| Model | Robust ADE | | Robust FDE | |
|:----------:|:----------:|:----:|:----------:|:----:|
| epsilon | 0.5 | 1.0 | 0.5 | 1.0 |
| Clean | 4.41 | 4.97 | 7.06 | 8.27 |
| *RobustTraj* | 2.54 | 2.73 | 4.47 | 5.43 |

### Q6: Limited discussion of related works
Due to the space limitation, only key prior arts are discussed in the Related Works section. In the next revision, we will expand on these discussions and add additional references to the existing topics. Please do not hesitate to point out works in other related threads of research that we did not consider.

#### References
[1] Wang, Jingkang, et al. "Advsim: Generating safety-critical scenarios for self-driving vehicles." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[2] Rempe, Davis, et al. "Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[3] Hanselmann, Niklas, et al. "KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients." arXiv preprint arXiv:2204.13683 (2022).
[4] Xie, Cihang, et al. "Adversarial examples improve image recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[5] Lee, Saehyung, Hyungyu Lee, and Sungroh Yoon. "Adversarial vertex mixup: Toward better adversarially robust generalization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[6] Tu, James, et al. "Adversarial attacks on multi-agent communication." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
[7] Zhang, Qingzhao, et al. "On adversarial robustness of trajectory prediction for autonomous vehicles." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[8] Croce, Francesco, and Matthias Hein. "Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks." International conference on machine learning. PMLR, 2020.

## Responses to Reviewer Bth3
### Q1: δ has box constraints. However for a purely waypoint based parameterizations this could suffer from poor realism, i.e. zig-zagging trajectories that are kinematically infeasible. Therefore, some of the attacks generated can be very unrealistic and out of distribution wrt real world data, and this could be a reason why vanilla adversarial training induces a regression in performance on clean data. I am very curious about enforcing realism into the threat model (for example using a bicycle model parameterization) and seeing if there is an improvement in the performance.
Thanks for pointing out this! We conduct an additional experiment to enforce realism into the threat model. Specifically, we conduct adversarial training by using the existing **search** method proposed by [1], named ***RobustTraj w/ search*** in the following table. [1] leverages the kinematic bicycle model to generate kinematically feasible adversarial trajectories. As shown below, *RobustTraj w/ search* is less robust than the *RobustTraj*, while there are minor performance improvements on clean data. We guess the potential reason is that although **search** [1] generates kinematically feasible adversarial trajectories by adding constraints to speed/acceleration/steering angle, it also reduces the attack effectiveness (as shown in Table B of Appendix). Therefore, the less effective attack during the adversarial training results in weaker model robustness against adversarial attacks while preserving performance on clean data at a certain level.

> Evaluation results of the adversarial training using *RobustTraj w/ search* [1].

| Model | ADE | Robust ADE | FDE | Robust FDE |
|:----------------------:|:----:|:----------:|:----:|:----------:|
| epsilon | 0.5 | 0.5 | 0.5 | 0.5 |
| Clean | 2.05 | 6.86 | 4.41 | 13.08 |
| *RobustTraj w/ search* | 2.10 | 4.75 | 4.37 | 8.35 |
| *RobustTraj* | 2.14 | 3.69 | 4.36 | 7.10 |

### Q2: Based on my understanding, the authors propose the deterministic attack as an alternative to sampling a random latent in the inner minimization problem of the threat model. It could also be important to consider the possibility of optimizing the latent to make the attack faithful to the formulation.
We do consider optimizing the latent. In Table 2 of the main paper, we conduct an ablation study considering three different attacks including the *latent attack*. The results indicate that the adversarial training with the latent attack results in less robust models compared to the proposed deterministic attack since the latent attack is a less effective attack (shown in Figure 4).

### Q3: Impact on downstream planners was considered in a close-loop setting however only qualitative examples were shown. I believe the results would be much stronger if quantitative results were presented for downstream planning.
Since attacking the prediction+planner is a time-consuming process, we only generated 10 adversarial scenarios. Among such 10 scenarios, all of them result in collisions, so we didn’t include the number. We will add more instances in diverse settings in the revised version.

#### Reference
[1] Zhang, Qingzhao, et al. "On adversarial robustness of trajectory prediction for autonomous vehicles." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

## Responses to Reviewer Bth3
### Q1: Lack of generalizability? The proposed adversarial training method may not be feasible/necessary for trajectory prediction methods like LSTM which directly predict future trajectories without sampling.
Thanks for your question. Most SOTA methods for trajectory prediction employ probabilistic generative models because they need to cope with the natural uncertainty of motion forecasting. Thus, our paper focuses on analyzing and tackling challenges emerging from attacking such probabilistic models.
We demonstrate that the proposed method generalizes to two mainstream probabilistic generative models: VAE-based and GAN-based models.
For the LSTM-based method which directly predicts future trajectory without sampling, their performance is behind the SOTA probabilistic generative models. As such, it is not the main goal of this paper. We will clarify the main scope of this paper in our revised version and leave attacking the LSTM without sampling as future work.

### Q2: If it can be tested in the ArgoVerse motion forecasting dataset too.
Thanks for pointing it out. We have tried our best to add results for this dataset. However, due to the negligible engineering effort for adapting a new dataset for the models (AgentFormer only support nuScenes) and limited computational resources, we are not able to provide the results given the limited time of rebuttal. We’d love to include that in the future.