Rebuttal
Reviewer cgp7:

1. re: In the related work section, the differences or advantages of this benchmark w.r.t. other benchmarks should be highlighted.
2. As far as I understand, there is no way to specify multiple constraints at the same time.
3. In Sec 4.3 (task 3 - robots with goal), what is the constraint?
4. While the various types of tasks are clarified, the paper also claims that it supports 33 different task settings. Could you please clarify what these 33 task settings are?
Answer:
problem1: To our best knowledge, the learning framework of safe RL: openai/safety-starter-agents, but this framework is based on TensorFlow1 and depends on gym 0.15.3, which is not conducive for RL researchers to use it. Safepo is a unified multithreaded PyTorch framework that allows different algorithms to be fairly compared on the same setting (e.g. same data preprocessing, same update frequency).
problem2: 我可以增加safety gym multi-constraint的实验

problem3: robot with goal Move the robot to a series of goal positions. When a goal is achieved, the goal location is randomly reset to someplace new, while keeping the rest of the layout the same. The sparse reward component is attained on achieving a goal position (robot enters the goal circle). The dense reward component gives a bonus for moving towards the goal. And the costraint is Hazards which are dangerous areas to avoid. These are circles on the ground that are non-physical, and the agent is penalized for entering them
#todo 论文没有加goal的实验，准备加上去，同时准备增加button实验

problem4: Safepo can support Safe-Gym, Bullet-Safe-Gym, as well as our newly designed 4 safe tasks on hands, with a total of more than 30 tasks.

Reviewer KqDf
1. The paper should highlight specific novel and contribution beyond the current benchmarks. While preexisting algorithms and benchmarks are mentioned in the Related Work section, there is no direct comparison of how SafePO provide in addition to the prior arts, such as between the SafetyGym and SafePO. 
2. line 48-50 (and corresponding section in the analysis part): "But in general, the approaches implemented by a deep neural network (e.g., P3O) achieve a better performance than the convex approximation method (e.g., CPO); such an empirical observation is sharper when the task is more complex ": could improve if the authors clarify on the criteria they considered for performance and how the task complexity is defined
3. line 59-61: "We observe that it is very challenging for the agent to learn a safe policy if the cost is too sparse.The empirical result shows that: the agent learns little to improve the reward performance and always prefers to play an unsafe policy. " could improve further if a clear description of the cost ranges that pose a challenge

problem1：
problem2：
When the state space and action space are large, CPO cannot guarantee the basic guarantee of SAFE in the convex approximation setting.
这边我会单独增加cpo的实验验证。

problem3:
In the current simulation environment design, a dense reward will be used to assist RL learning, and a large task bonus will be given when the final target task is completed. For example, if we want an agent to go to a certain destination, we will not only give a reward for reaching the destination, but also give a reward design that is closer and closer to the destination, which is more conducive to the learning of the agent. When we only give the agent a reward for completing the task, when the state space and action space are large, it will make the learning process very challenging. In the safe RL setting, when our cost penalty signal is too sparse, it will also make it difficult for the agent to learn a safe policy.

re: 59Ne
As a benchmark paper, the performance of various baseline algorithms is critical. Unfortunately, the results shown in table1 and table3 are far worse than those reported in previous papers. If the baseline algorithm does not have the desired effect, the conclusions based on the experimental results are meaningless. At the same time, the following RL researchers will not benefit from it.
What did the previous article refer to? Is it in the same setting as ours? It may be that our cost setting is different from other parameter Settings. For example, Focops has published its own code on Github, but when we took the code, we verified that it could not match the experimental data in its paper in Ant-v3 and other environments. At the same time, we modified it to the environment of Safety Gym, only modified the way of interaction between the model and the environment, and the algorithm result was also very poor. We also found someone who had the same problem as us in the Github issue, and the author replied that it was necessary to adjust the parameters carefully. However, there are still some problems. When we did the experiment, we found that the author's original implementation method, single-threaded mini-batch, needed 10h in safety gym in a simple environment, which was very unfriendly to RL researchers. Therefore, we made SafePO, which has a more efficient sampling speed and a more unified framework. Let RL researchers fairly compare the performance of different algorithms under a unified frpamework.

Paper present four safety robot tasks, including robots with limited velocity, robots with circle, robots with goal and safety dexterousHands. However, these environments have been proposed in previous works, and the author's contribution is to re-test the implemented baseline algorithm in these tasks. Based on this, it is difficult for me to identify this as the main contribution of a paper.
The Datasets and Benchmarks track serves as a novel venue for high-quality publications, talks, and posters on highly valuable machine learning datasets and benchmarks, as well as a forum for discussions on how to improve dataset development. Our innovation is to create a unified algorithmic framework for Safe RL researchers, which is the starting point of our work. At present, there is no unified framework for fair algorithm comparison in the Safe RL field. For example, the code of PCPO is still Theano. When implementing PyTorch, we are very careful to follow the original text and make a careful comparison with Theano code to ensure that there are no errors. The algorithm code for P3O has not been published, etc. I think our work has played a great role in promoting the follow-up research on Policy Optimazation in the whole Safe RL.

The analysis of the results for various baseline algorithms is insufficient. Paper does not explain why the algorithms fail on some safe tasks. In Discussion C.3, the authors attempt to analyze the difference between (FOCOPS, PPO-L, and P3O) and (CPO, PCPO, and TRPO-L). However, these discussions are not linked to the experimental results.

re: hb7D
There are grammatical errors and issues with flow throughout. There are many places in the paper where the language is non-technical or unusual, e.g., Lagrange multiplier “keeping calm” (lines 58, 225), calling an algorithm a “winner” (lines 181, 182), and “solid” safe RL benchmark (line 7).
The paragraph in lines 188-198 is incredibly confusing and does not help me understand why CPO, TRPO-L, and PCPO are underperforming.
Section 6.4 (Sparse Cost) is also confusing. “This is because the total cost limit is tiny, so the robot obtains a small cost for each step, implying the robot knows very little about the environment.” This statement makes no sense. Why would a small cost limit result in the robot getting a small cost in each step? How do we know the robot “knows very little about the environment”? “We know the algorithm near the cost limit usually obtains better performance than that far away from the cost limit. This is because the higher cost implies the robot learns more about the environment, which helps the robot understand a safer policy.” Why would a higher cost imply that we’ve learned more about the environment? Why does a policy “near the cost limit” imply it performs better? These are unsubstantiated and (in general) false claims.

re: fKDX
[W1] The proposed benchmark suite is mostly a collection of existing environments (Limited Velocity Robots, Safety-gym, Bullet-safety-gym) and re-implementations of existing algorithms. It, therefore, has limited novelty. I'm confused about the main contribution of this paper.
The Datasets and Benchmarks track serves as a novel venue for high-quality publications, talks, and posters on highly valuable machine learning datasets and benchmarks, as well as a forum for discussions on how to improve dataset development. Our innovation is to create a unified algorithmic framework for Safe RL researchers, which is the starting point of our work. At present, there is no unified framework for fair algorithm comparison in the Safe RL field. For example, the code of PCPO is still Theano. When implementing PyTorch, we are very careful to follow the original text and make a careful comparison with Theano code to ensure that there are no errors. The algorithm code for P3O has not been published, etc. I think our work has played a great role in promoting the follow-up research on Policy Optimazation in the whole Safe RL.

[W2] The analysis of the experiment is not deep enough in the paper, authors largely rehash existing ideas. (Sec 6.3 Sec 6.4).

[W3] The authors mentioned the sparse cost problem in sec 6.4, but they do not discuss it more and give possible solutions.
这边的解决方案是增加多一些关于sparse cost 的实验

[W4] In the provided code, the safe-marl-baselines algorithm seems to be derived directly from [1], which is not properly integrated into the SafePO framework.
这个是一个问题，需要讨论