## New experiment

We perform additional experiments on CartPole-v1 environment from OpenAI gym. To evaluate the risk-preference performance of the learning method, we modify the environment by perturbing the reward with zero-mean random noise when the cart enters into some certain regions. A risk-aware policy is capable of avoiding such regions without compromising the cumulative return performance. The maximal episode length is $500$, and the cart position is randomly initialized near the central point.

We set the perturb region to $x<-0.1$, where x is the cart position. The reward is perturbed by $\delta\sim \text{Uniform}[-6, 6]$ as long as the cart stays in the region. The policy is parameterized by a multilayer perceptron with two hidden layers of size (64, 64). We average all results over five random seeds. While both LSD and REINFORCE are able to achieve the optimal expected cumulative return of $500$, the policy learned by LSD is able to steer the cart away from the perturbed region (see figure (b)). This leads to a more concentrated cumulative return distribution (see figure (c\)) that stochastically dominates that induced by the policy learned by REINFORCE (see figure (d)).

The figures can be viewed with the following anonymous link.

([image link](https://ibb.co/C8Bktjb))

## Reviewer kgwT

Thanks for your time reviewing our paper and for your feedback.

> Theoretically, is the defined $\theta^*$ good in terms of stochastic dominance?

> While the motivation of the submission is to learn some solution "good" in terms of stochastic dominance. However, the authors change this objective to learning a $\theta^\star$. Therefore, a natural question is whether the $\theta^\star$ is good in terms of stochastic dominance? For example, does $\theta^\star$ stochastic dominants any given other solutions with high probability? This is a crucial question of the framework of the paper. If $\theta^\star$ does not match to a "stochastic dominating" solution. The goal of the optimization method does not make sense any more.

We would like to clarify that this work has been focusing on the sole goal of learning a "good" solution from a candidate solution set in terms of stochastic dominance. For parameterized models, this objective is *equivalent* to picking an optimal parameter $\theta^\star$, such that the induced $X_{\theta^\star}$ is not dominated by $X_\theta$ for any other choice of $\theta$. Specifically, in Theorem 1, we show that this goal can be approximately achieved under the proposed generalized stochastic dominance metric $\Omega$.

> Is there a better evaluation metric for the experiment results?

> Currently, from the presented experiment metrics, the performance improvement of LSD seems to be small. This actually makes sense, as the evaluation metrics considered as still existing ones like accuracy, CE loss, and variance: they are not consistent with the stochastic dominance. Can you provide some metrics that are more consistent with the stochastic dominance to show that the proposed method can really learn a solution that is more "stochastic dominating" other solutions?

> For example, can you show that the proposed method learns a solution that stochastic dominates the solutions learned by existing methods?

We illustrate in Figure 2 that in a simple reinforcement learning task that the proposed method learns a more conservative policy that indeed stochastically dominates the one learned by the standard policy gradient method (REINFORCE). Similar results have been also achieved in our new experiments on the CartPole-v1 task. As shown in subfigure (d), the second order CDF of learned policy's cumulative return distribution consistently stays below that of REINFORCE method, indicating that the proposed method can really learn a solution that is more "stochastic dominating" than other solutions.

While it may sound natural to quote $\Omega$ as an universal numeric evalutation metric in the experiments, there are two reasons that keep us away from that: i) the metric fail to distinguish non stochastically dominated solutions, e.g., the solutions for comparison in the portfolio optimization task where the Sharpe ratio serves as a more reasonable criteria; and ii) in supervised learning tasks, a neural network overfitted to the test set, though not learnable from the training data, is able to stochastically dominates any learned solution and thus makes $\Omega$ less meaningful.

## Reviewer 3Lzu

Thanks for your time reviewing our paper and for your feedback.

> I wonder what is the connection between SD and GSD. This new version of SD only guarantees the optimality for $\eta \in [a, b]$. Will this affect the applicability of this new notion?

The proposed generalized stochastic dominance functional admits $X\succeq_k Y \iff \Omega(X, Y) := \max_{\eta}[F_k(X;\eta) - F_k(Y;\eta)] \le 0$. It is named 'generalized' in the sense that $\Omega$ quantifies the degree of stochastic dominance for an arbitrary pair of random variables $(X, Y)$ and therefore enables designing a practical algorithm.

While it may sound counterintuitive, we remark that the optimality for $\eta \in [a, b]$ actually implies the optimality for a larger interval $[c, d]$: if $X_{\theta^\star}$ is not dominated on $[a, b]$ by any $X_\theta$, then $X_{\theta^\star}$ cannot be stochastically dominated on $[c, d]$ as well if $[a, b]\subset[c, d]$. As different choices of $a$ and $b$ lead to different solution sets, this grants the new notion the versatility to focus on risk preference with regard to a more selective range of values.

> In practice, what will be a good choice of $k$?

The most common choices in practice are $k=1$ and $k=2$, as the stochastica dominance rules can be interpreted as focusing on all increasing (resp. concave and increasing) utility functions in the expected utility framework for $k=1$ (resp. $k=2$).

## Reviewer vGkD

Thanks for your time reviewing our paper and for your feedback.

> The evaluation of supervised learning is based on simple dataset (the method also doesn't achieve the best accuracy). Could the authors investigate on larger dataset/network to provide a whole picture? Likewise, for reinforcement learning, could the authors compare with more state-of-the-art algorithms and perhaps a few more environments for diversity? Overall, I think a more comprehensive empirical evaluation is needed to convince the readers that the proposed method is effective.

As the reviewer suggested, we have included additional experiments on the CartPole-v1 task from OpenAI's gym, where the policy learned by the proposed method is able to avoid risky regions without explicitly applying constraints/penalties and achieve the optimal cumulative return. We remark that as REINFORCE has already achieved optimal cumulative return performance on these tasks, the learned policy is indistinguishable from that learned by state-of-the-art RL algorithms, and is thus sufficient to verify the efficacy of the proposed method in enabling risk-sensitive decision making. That being said, it is an important direction for future research to develop a more streamlined and scalable RL algorithm capable of solving more complicated tasks. For supervised learning, we plan to include more experiments in the revision as well.

<!-- [can be removed] Thank you for your suggestion on adding more comparisons and environments to improve the diversity of the experiments, which could really help make the results more convincing. As suggested, we added another environment CartPole-v1 from OpenAI gym. The results can be found in the anonymous link, and they show that the learned policy's value distribution dominates that of REINFORCE (please see subfigure (d) in the link), which verify the proposed method. The amount of experiments is comparable with existing study on distributional RL (for example https://arxiv.org/abs/1905.07318), given that our method is beyond RL and applicable to also supervised learning, which is a sign of sufficiency of our study. -->