# ICLR 2023 relay rebuttal

| Game | Same Agent Failure Rate <br /> = P(len(traj)<500\| len(traj)> 400) | Failure Rate of <br /> Relay Evaluation |
| -------- | -------- | -------- |
| heist | **2.02 $\pm$ 0.46** % | **9.54 $\pm$ 0.95** % |
| jumper | **4.96 $\pm$ 0.72** % | **11.59 $\pm$ 1.04** % |
| bigfish | 5.51 $\pm$ 0.79 % | 5.82 $\pm$ 0.82 % |
| ninja | **10.60 $\pm$ 1.04** %| **22.07 $\pm$ 1.35** % |
| leaper | **11.38 $\pm$ 0.96** % | **26.02 $\pm$ 1.32** % |
| dodgeball | **10.29 $\pm$ 1.05** % | **15.06 $\pm$ 1.23** % |
| bossfight | 24.56 $\pm$ 1.34 %| 25.07 $\pm$ 1.34 % |


<!-- | bossfight | 75.44 $\pm$ 1.34 | 74.93 $\pm$ 1.34 |
| starpilot | 53.26 $\pm$ 2.18 | 53.27 $\pm$ 2.16 |
| chaser | 32.29 $\pm$ 1.69 | 32.45 $\pm$ 1.70 |
| fruitbot | 0.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | -->


## Reviewer vBmj

Dear Reviewer vBmJ,
We really appreciate your constructive comments. We believe the “gap” between the experiments and the claimed application scenarios, as you mentioned, is partially caused by a misunderstanding. We will address each of your questions and concerns below:

> "a self-driving system may need to take over the control from humans in the middle of driving and must continue to drive the car safely.”

In this example, the self-driving system is the test agent, and the humans are the stranger agent. Humans are strangers to the self-driving system since it is hard to fully present the "driving style" of a particular user when training the agent. However, the self-driving system still needs to be general enough to safely take over control from a state produced by an unknown human driver. Our relay-evaluation is one way to evaluate agent performance starting from the states that it may not see during training.

> Agents using *different* RL algorithms are considered the main part of the experiments…there exists a gap between the experiments and the claimed application scenarios

We consider stranger agents trained using **both the same and different** RL agents because a wider range of agents will cover more controllable states and leads to better evaluation of generalization, hoping to **reduce the gap** to realistic application scenarios. In the human driver example, although it is difficult to exactly model human behavior, if we can cover as much space of controllable states as possible during evaluation (e.g., using states generated from multiple RL algorithms), we are more likely to cover the states that human drivers will produce.

> I wonder does there exist a gap between the experiments and the claimed application scenarios.

Because human behaviors are hard to model, we agree there may exist a gap between the experiments and *real-world* application scenarios since we only consider relay-revaluation of a diverse set of RL agents, not humans. The modeling of human behavior is much harder and beyond the scope of this work. Our main contribution is to provide a cheap and practical way to evaluate relay-generalization as the first step to solving the real application scenario. In addition, taking over the control from other RL agents (not just humans) also has real application scenarios. For example, assume we have driving agents trained for different goal functions such as speed, gas efficiency, or comfortability. A user may want to switch from the most comfortable agent to the fastest agent since he is running late.

> Can authors compare their works with several popular meta-RL frameworks?

We have added a discussion of meta-RL in Related Work. The goal of relay generalization is different from meta-RL. In our work, the test agent is only trained for **one** task, and we evaluate if the test agent can complete the task starting from a stranger agent’s trajectory. In Meta RL, people normally first train an agent for task A and use the prior knowledge learned in task A to train another agent for task B. On the one hand, our method does not require the agent to do another task. After the test agent takes over the control, its goal is still to complete its original task. On the other hand, we aim to study the generalization to a "stranger agent's trajectory." Thus the agent **cannot know** the stranger agent's trajectory beforehand, and **cannot be fine-tuned** according to the stranger agent's trajectory.

> So, I begin to question the performance of STA, maybe the simple way in the naive algorithm can already handle relay evaluation?

The biggest difference between STA and naive is the **training cost**. For the naive method, if we pretrain K agents, then it will increase the training time (and the number of environment interactions) by K+1 folds, which is usually impractical. In contrast, STA barely increases the training time compared to vanilla SAC. For example, the time for training SAC in Humanoid is about 130,000 seconds, while the training time of STA is about 140,000 seconds. Moreover, in the hardest environment – Humanoid, STA is consistently better than the naive method by a significant margin. For example, if the stranger algorithm is SA-PPO, the failure rate of STA is 25%, while the naive method is 58%.

> Reproducibility

To increase the reproducibility of our STA algorithm, we have rewritten section 3.2 with the pseudo codes provided. We also upload our code anonymously ( https://anonymous.4open.science/r/spinningup-E8B1). The implementation of STA is in ./spinup/algos/pytorch/gsac/gsac2.py. We will open-source our project after our paper is accepted.

We thank the reviewer again for your insightful comments, and we hope we have addressed your concerns on the motivation, connection to meta-RL, and the differences between native and STA agents.

Sincerely,
Paper 6011 Authors

## reviewer KKAE

We thank the reviewer for the clear summarization and all the constructive suggestions. To address your concerns, we conduct additional experiments to show that the lack of generalization in relay-generalization is persistent under different failure criterions and infinite replay buffer size. We also demonstrate that our STA agents indeed generalize better to states they did not find during training.

> (Q3) It is possible the termination criteria in the MuJoCo environments are too strict and terminate policies that might eventually succeed.

Thank you for pointing this out this possibility. To understand the severity of this problem, we conduct the following experiment. Instead of terminating the agent when it is flagged as "terminated" by the environment, we disabled termination in environments and let the agent runs to the maximum number of steps anyway and redefines "failure" by the return and a threshold. If the agent gets a return lower than the threshold, then we define the agent fails. Since the expected return of agents in standard evaluation are all greater than 2800, we use three different thresholds (500, 1000, 1500) to define failure in our experiment.

The results are shown in [Table 10](https://i.imgur.com/AipVY1U.png) in Appendix G and Table 10. According to the results, there did exist trajectories that eventually succeeded. However, the failure rates are still unacceptably high. For example, even if we define having a return that is higher than 500 as success, the fail rate on Humanoid SA-PPO is still 81.83, which is not much different from Mujoco’s definition. In general, we are only trying to provide the general evaluation framework for relay-generalization in this paper, and the definition of "failure" may depend on the use cases and the environments.

[Link to Table 10](https://i.imgur.com/AipVY1U.png)
> (Weaknesses) How much of an issue would this be with fully off-policy methods with large replay buffers?

Both SAC and TD3 are off-policy methods with large replay buffers. Their replay buffer sizes are one million, while the total number of environment interactions is only three million. To understand if increasing the replay buffer will help, we train the SAC agent with an infinite replay buffer. That is, all the trajectories will always be in the replay buffer. The results are in [Table 8](https://i.imgur.com/9R4AJqR.png) in Appendix D.

According to the results, SAC with an infinite replay buffer still has poor performance on relay evaluation compared to our method. Moreover, the average return under ordinary evaluation is lower than the vanilla SAC, while our method is greater or equal to the vanilla SAC.

> (Weakness) it is unclear how much it can improve generalization performance overall, for example, from states that no policy would be likely to find during training

During STA training we do not use the agents for relay-evaluation (no leaking of test data); only 1 STA agent is trained using SAC.The trajectories used in relay-evaluation are generated by many stranger agents, and it's very unlikely that they appear during STA training. The purpose of this paper is to evaluate agent performance on a diverse set of valid and controllable states (much larger than standard evaluation) that are unlikely to be seen during training, and the good performance of STA shows that it can improve generalization performance in this setting. Indeed, we cannot evaluate generalization to states that no stranger policy can generate. However, to our best knowledge, no previous works can evaluate this type of generalization in complex environments because of the hardness of finding valid and controllable states for evaluation. Our paper has made a major step towards better generalization evaluation via relay evaluation.

For empirical evidence, to show that the states of the stranger agents are indeed unseen to our STA agent during training, we use t-SNE to visualize the historical states of an STA agent during training along with the states from 6 stranger agents. The figure is shown in [Figure 3](https://i.imgur.com/ZY0lENZ.png) in Appendix H. According to Fig. 3, although the distribution of historical states of the STA agent is more diverse than the distribution of other agents’ states, there are still many areas that it does not cover. This suggests that the STA agents have a better performance on relay evaluation due to being more general, not just memorizing its training states

[Link to Figure 3](https://i.imgur.com/ZY0lENZ.png)

> (Weakness) the solution STA is relatively simple

As the first paper that tries to define and solve the relay-generalization, we hope our method can serve as a simple and intuitive basic method and could be widely used in future developments of potentially more complicated algorithms, just like data augmentation in image classification.

> (Q4) Definition 2.1 is not entirely clear.

Thank you for bringing up the question. We do require valid actions to move to the next states (no jumps, it’s not an arbitrary state sequence), although we focus on states and not actions. We have revised the definition to make it more clear in the paper by including the actions in our definition of a trajectory.

> (Q3) The "Failure" criteria should be described for each environment

Thank you for the reminder. The criteria defined by OpenAI Gym Mujoco environments are now listed in Appendix F.

> (Q5) it is not explicitly stated that the "stranger" and "test" policies are trained using the same configuration for the same number of steps.

Different algorithms use different configurations. For off-policy algorithms like SAC, TD3, the naive method, and STA, we use the same configuration. For example, they all trained for three million interactions. For on-policy like PPO, SA-PPO, and ALTA-PPO, they are all trained for one million interactions. For other parameters of the on-policy algorithm, we follow the fine-tuned parameter of those papers’ GitHub. More details are in Appendix A.

> (Q2) Table 1 should show returns

As you suggested, we list the returns of the agents in [Table 13](https://i.imgur.com/EbiECNJ.png) in Appendix K.

[Link to Table 13](https://i.imgur.com/EbiECNJ.png)

> (Q1, Q6) Typos, related work section is a long block

We have fixed the typos and divided the related work section into a few points. Thank you for the suggestion.

## Reviewer 9fm8

Thank you for the positive and encouraging review! We address your concerns below.

> Reproducibility

To increase the reproducibility of our STA algorithm. We have rewritten section 3.2 with the pseudo codes provided. We also upload our code anonymously ( https://anonymous.4open.science/r/spinningup-E8B1). The implementation of STA is in ./spinup/algos/pytorch/gsac/gsac2.py. We will open-source our project in the future.

> Authors sampled states from trajectories generated by stranger agents, which are trained with only different random seeds.

We also sampled states from trajectories generated by stranger agents, which are trained with different **algorithms** and different seeds.

> PBT[1] might be of help in finding more "controllable" states

Thank you for sharing this interesting paper, we have cited it in our revision.
In our work, we have generated different policies using a diverse set of training algorithms, but we agree that it’s interesting to generate diverse policies using different hyperparameters. To study whether the diversity of hyperparameters helps, we conduct the following experiments. We train 40 SAC agents by choosing hyperparameters randomly and then use the top 10 agents to generate controllable states. The relay evaluation results on these controllable states are shown in [Table 12](https://i.imgur.com/DJIKINx.png) in Appendix J. According to Table 12, we observe that the states generated by random hyperparameters are easier for other agents to pass the relay-evaluation, especially when evaluating SAC and TD3 agents. A potential reason is that some of these hyperparameters lead to worse policies and generate slower trajectories, and it’s easier for the target agent to take over from those slower states.

[Link to Table 12](https://i.imgur.com/DJIKINx.png)

> Many components applied in STA are more likely intuitive.

As the first paper that tries to define and solve the relay-generalization, we hope our method can serve as a simple and intuitive baseline and could be widely used in future developments of potentially more complicated algorithms.

> a provable improvement might not be guaranteed

We agree a formal theoretical guarantee could be an interesting future work.
To experimentally show the consistency of our improvement, we not only test on different environments but also on different stranger algorithms. The results in Table 4 show that our method can consistently drop the failure rate dramatically.

## Reviewer 7NUq

We thank you for all the valuable comments and suggestions. We have **revised Section 3.2** according to your suggestions to improve clarity and added pseudo code to present our algorithm clearly. As the pseudo-code shows, STA is a very simple framework. In the following, we first conduct **several ablation experiments** to justify our choice of the components in STA. We then address concerns on relay-evaluations. Finally, we compare our work to Tandem RL.

### Questions about our STA training method

> (Weakness paragraph 4) Can the STA training scheme negatively impact the agent’s performance? Is it possible that the optimal policy for this augmented MDP is different from that of the original MDP?

Although the optimal policy for our augmented MDP may be different from the original MDP, the results in Table 4 show that our methods can usually improve the return compared to standard training. In Table 4, we show the results of the agent’s performance in the ‘ordinary’ rows. As the results show, the ‘Return’ column of STA and the naive method is greater or equal to the ‘Return’ column of SAC in all three environments. Hence, we believe that our method is **similar to data augmentation in computer vision, which can lead to a solution with better generalization and testing performance**.

> (Weakness bullet 1) Some states may not be included just because these bad policies can't perform well when starting from them.

We agree that some controllable states may not be included in $\mathcal{S}_{STA}$. During training, we need a quick estimation for the "controllability," so we only added states that can get a high reward in the next $\lambda$ steps. **It is unavoidable to miss some controllable states because it can be costly to enumerate all of them during training**; similarly, in data augmentation, we cannot add all possible ways of augmentation and rather just sample some augmented images. By adding these states (which may be a subset of controllable states), we hope the agent can be more generalizable and work well with other controllable states. According to our experiments, even missing some of the controllable states, STA can greatly improve the relay-evaluation results.

> (Weakness bullet 3) Why do the states need to be controllable to be added to the initial set

Our goal is to train the agent such that it can play well starting from controllable states instead of any state; **if a state is uncontrollable, then there’s no way to expect the agent to play well** starting from that state. Therefore we only add controllable states to the initial set.

> (Weakness bullet 3) What happens if you just add all states to the starting set regardless of whether they are controllable?

We agree that adding some uncontrollable states to $S_{STA}$ might be helpful. However, including too many uncontrollable states is not always a good idea. In [Table 7](https://i.imgur.com/oKArMUY.png), we show that if we further increase $\gamma$ (adding more uncontrollable states to $S_{STA}$ ), the Humanoid performance will drop, indicating adding too many uncontrollable states is not beneficial.

[Link to Table 7](https://i.imgur.com/oKArMUY.png)

> (Weakness bullet 4) Why is low $Q(s,\pi(s))$ a good metric to determine if the agent is not familiar with a particular state?

If the $Q(s,\pi(s))$ is low, it is either 1) the agent is bad at this state or 2) the agent is good at this state, but its Q function is wrong. **For both scenarios, more training is needed for the candidate state $s$**. Moreover, according to the ablation study in Table 2, given the agent is bad at this state, the $Q(s,\pi(s))$ is normally pretty low.

> (Weakness bullet 5) Parameter $\lambda$: Can't you just divide the score by the number of remaining timesteps?

Thank you for the suggestion. We agree that using the average reward (score divided by the number of time steps) is another option. To test this alternative, we conduct an experiment using the average reward of the remaining steps in [Table 9](https://i.imgur.com/9cSUeNK.png), Appendix E. According to the results, using the average reward can also reduce the failure rate compared to the original SAC. However, its failure rate is still higher than using our score function (column STA), which uses the sum of the reward of the next $\lambda$ steps.

[Link to Table 9](https://i.imgur.com/9cSUeNK.png)

> (Weakness bullet 2) Why don’t you use the Q network to estimate the score?

Using the Q value may also miss some of the controllable states since the Q value may also be wrong. Moreover, it might even add some uncontrollable states into $\mathcal{S}_{STA}$. Using a high return as the criterion ensures the added states are controllable.

> (Weakness bullet 2) Is the score of a state averaged over multiple runs?

Most of the state’s score is just single runs. Only the states that have been sampled from $\mathcal{S}_{STA}$ as the initial state will have multiple runs. For each of those states, the score is the maximum score of all the scores it has gotten in all the runs.
The score served as evidence showing that it is possible to get a high reward at the next $\lambda$ steps from the state. Although we might miss some of the states, the states we added are able to increase the relay-generalization of the STA agent.

> (Weakness bullet 6) How is $\eta_{\text{STA}}$ computed?

It is selected by experiments of using different $\eta_{\text{STA}} = 0.5, 0.75, 0.9, 0.95$.

> (Weakness paragraph 5) Simulator can be reset to any state

Although resetting to any valid state is possible for most environments (e.g., by setting physical states of objects, or game memory and program states), if simulators do have restrictions, we can include only those states that are both controllable and "resetable".

### Questions of Relay-evaluation

> (Weakness paragraph 1) Evaluation is computationally heavy it requires training multiple other agents to ensure most of the test and stranger agents are not very similar

In our experiments, **failure is very common**. It even happens on the agents trained with the same algorithms as stranger agents. According to our results, all agents in Humanoid have at least a 10% failure rate on any other agents' trajectories, so one can observe the problem of relay generalization even when sampling only one agent. We evaluate each agent by multiple (4*10) other agents **just for a thorough and comprehensive comparison** and to reduce the variance in reported results, and users can choose the number of agents according to their budgets. In addition, we will ​publicize the **states of stranger agents as a dataset** so that people can reproduce our experiment easily without training a large number of agents

> (Weakness paragraph 2) Definition of controllable state: what does “high return” mean

In definition 2.1, we do not explicitly define “high reward” since **it depends on the reward structure of a particular environment**, and we want the definition to be generic. In our experiments, we defined “high return” as if the return of a trajectory is higher than the top $\eta$ ratio of all the trajectories. According to our internal experiments, the result is insensitive to this definition, and the observations are almost the same when $\eta \in [0.25, 0.75]$. Hence, we just simply select $\eta=0.5$.

> (Weakness paragraph 2) how large does L need to be

In our setting, the maximum steps of the environments are “$L_{max}$=1000”. If L is too small, the agent might pass the relay-evaluation by luck. If L is too large, we will have fewer states to sample since we will only sample $s_t$ that $t<L_{max}-L$. Hence, we select $L=500$. We also conduct experiments with different $L$ in [Table 11](https://i.imgur.com/c4rt8Op.png) in Appendix I. The results show that most of the failures happened in the first 100 steps after the test agent took over the control. The results also show that for some of the cases, even if the agent does not fail in the next 100 steps, it still has a chance to fail in the next 400 steps.

[Link to Table 11](https://i.imgur.com/c4rt8Op.png)

### Tandem RL

> “The Difficulty of Passive Learning in Deep Reinforcement Learning”

Thank you for sharing this interesting paper. We have cited this paper in our revision.
Tandem RL shows that it is hard to **train** an offline RL agent even if the agent can observe the historical trajectory of another agent and follow all the training processes. On the other hand, relay-evaluation shows that during **evaluation**, it is difficult for a well-trained agent to start from a state of another agent’s trajectory.

### Typos

Thank you so much for carefully reading our paper and we have fixed those typos. Thanks!