# Response to Reviewer \#1
You may have some misunderstandings about our paper.

About the topic of our paper: In line 076, we quote a possible explanation for style and accordingly define style with the pattern of wording (namely a set of keywords). The main purpose of our paper is to put forward and formalize the task of stylized story generation, and to propose a baseline model for the new task. As you can see, the method to extract the keywords from texts is not the main concern in our paper. Therefore we choose two common styles (emotion and event), whose keywords are easy to extract using off-the-shelf tools, to illustrate the task and validate our model. We are not reducing style to emotion. In line 094 we mentioned that styles can be extended by defining new stylistic keywords.

About the automated evaluation metrics: Since we have proposed a new task. there need to be some metrics for evaluation. Therefore, the logic is that we put forward LSC and SSC according to the definition of style and then further validate them using human evaluation. The results show our automated metrics have high consistency with human evaluation. yao2019 also uses the same paradigm in the aspect of evaluation.

About human evaluation: In line 380, by saying "we hire three annotators ...", we mean for each pairwise comparison we collected the opinion of 3 annotators, instead of completing all the comparisons with only 3 people. As for subjective interpretation, we specify the meaning of fluency, coherence, and style consistency for participants, which can be seen in the appendix (line 257 ~ line 280). The kappa results also show that the annotators highly agree with each other. Therefore it seems that human evaluation is not affected by subjective interpretation. We are sorry that we did not know about Hämäläinen et al. (2020) at the time so we choose to follow the pairwise comparison paradigm in yao2019 for human evaluation. Moreover, we introduce fluency and consistency in evaluation to illustrate our model is not copying keywords into stories without considering their contextual meaning. The improvements of these non-stylistic features should be due to the introduction of the planning procedure.

# Response to Reviewer \#2

Answer-to-W1: It is a misunderstanding due to our vague wording. In our scope of the survey, we did not find papers that investigated stylized story generation previously.

Answer-to-W2: Your suggestion is really helpful. We should not limit the definition of style in terms of keywords. We can give more general definitions while emphasizing the definition is keywords in our work.

Answer-to-W3: If we decode keywords explicitly from the keywords distribution, the two-step method is non-differentiable. Equivalently, we have to train two separate models, which is called pipeline. The first model (keywords planning model) generates keywords distribution and decode keywords from that distribution. The second model (story generating model) generates stories base on keywords and the given context (the first sentence and the given style). However, the keywords used in the story generating model when training are golden truth while the keywords used in inference are from the output of the keywords planning model. The difference between training and inference introduces bias.

If we do not decode keywords explicitly and use the distribution directly, the whole generating process is differentiable. Thus, it can reduce the bias mentioned above. In short, using distribution instead of decoding makes our model end-to-end.

Answer-to-Q1: To some degree, there is a similarity between LSC and SSC metrics. In our consideration, the SSC is more intuitional and can learn other factors apart from keywords that determine the style of the story.

If a story consists of stylized keywords but lacks coherence, the SSC can be much convincing than LSC. This can be achieved by introducing some negative samples when training the separate model using in SSC.

For example, a story is "Happy is happy. I am very happy. Happy can be happy. Everyone is happy.". It will get high scores in emotional LSC, but apparently, it is not a fluent emotion-driven story and supposed to be classified into <other>. If we train our SSC model with this kind of samples, SSC will be more convincing and reasonable.

I think this learnable automatic metric (SSC) we emphasized here can be an inspiration for future works.

# Response to Reviewer \#3
Answer-to-W1
Our proposed method's major difference between previous techniques (such as Yao2019) is that we do not generate keywords from the distribution explicitly. Instead, we use the keywords distribution directly. In Eq(5) of section 3.3, we combine the language model's output P_l with the keyword distribution P_k. This also explains why our model's structure is different from previous works. Our proposed two-steps model is entirely differentiable and end-to-end. Previous techniques (like Yao2019) treat the two steps detached to some degree, because they decode keywords from the distribution and makes the process non-differentiable. Back Propagation algorithm cannot work with this decoding strategy. In short, our model is end-to-end and easier to train, while previous techniques (like Yao2019) is pipeline.

Answer-to-W2
Our model adds a planning module to Bart.
Therefore the comparison between our module and Bart can be seen as ablation study for the planning module.

Answer-to-W3
There is no available dataset for Stylized Story Generation task, because it is a new task we proposed in our work. Many of the stories in the existing datasets do not have obvious stylistic feature because they are not meant for this task. Constructing a high-quality dataset for this task may be leaved for future work.

Answer-to-W4
Due to space constraints, we do not show our comparsion with previous story generation methods or ideas. We will present the experiment of comparsion with Yao2019 model in the final version of paper.

Answer-to-Q1
Our proposed method's major difference between previous techniques (such as Yao2019) is that we do not generate keywords from the distribution explicitly. Instead, we use the keywords distribution directly. The keywords distribution from keywords planning module will combine with the output of the language model (a probability distribution on the whole vocabulary), which is formulated in Eq(5) in section 3.3. This also explains why we generate the keywords distribution on the whole vocabulary, instead of on the stlized vocabulary.

The reason why we use keywords distribution directly is that we can make the whole process differentiable. If we decode the keywords from distribution explicitly, the derivative cannot be back propagated there. However, using the distribution can solve this problem. The two steps (keywords planning step and generating with keywords distribution step) are integrated and differentiable.

Answer-to-Q2
Our methods is split into two main parts:

1. Generate keywords distribution from given context(first sentence and the given style)
2. In each word prediction step of language model, we will use the keywords distribution to adjust the output from langugae model.

Thus, when generating keywords distribution in part 1, we can only take it as bag of words, since it is a distribution. The sequence infomation will be reflected in part 2, since every word will be affected by the keywords distribution.

# Response to Reviewer \#1

You may have some misunderstanding on our paper.

In line 076, we have quoted a possible explanation for style and accordingly defined style with pattern of wording (namely a set of keywords).
The main purpose of our paper is to put forward and formalize the task of stylized story generation, and to propose a baseline model for the new task. As you can see, the method to extract the keywords from texts is not the main concern in our paper. Therefore we cho

topic：在introduction有引用风格的定义，我们在本篇文章中是用词语的使用pattern来定义风格的。我们的目的在于formalize这个任务并且提出一个相对有效的baseline模型，而emotion和event两种“风格”只是作为用来验证的例子，选取这两个作为例子的原因是因为emotional的关键词和以及event相关的关键词（在文中以实义动词定义）有现成工具可以提取。这种定义可以扩展到别的风格，例如想要”魔法“的风格，那关键词的词表可能就是”魔杖“、”咒语“等词汇，但这样的关键词通常需要人工抽取，而我们的论文的关键点并不在如何抽取关键词。我们只是用emotion和event两种关键词相对容易抽取的风格来阐述我们所定义的任务并验证模型的有效性。

基于我们所给出的风格定义，基于词汇pattern定义评测metric是比较合理的选择，我们也通过human evaluation说明了这种评测方法一定程度上是合理的。为了能有更直白的评测方法，也为了让评测结果对语义进行考虑，我们也通过bert训练了分类器进行评测，即文中说的SSC（例如无限复制stylic words，用LSC仍然会得到很高的结果，但SSC会因为它不符合语言规则而分类到normal中）。

对于human evaluation部分，可能有一些误解。我们是指对每一条数据会有三个人进行人工评价，而不是说总共只有三个人进行了人工评价。

# Response to Reviewer \#2

SSC可能会把一些瞎复制关键词的故事分成normal（这个是不是得做实验看一下）

语法错误会改的

感谢您的肯定

# Response to Reviewer \#3

1. planning
The novelty of our model is

3. ablation study其实就是再加一个和yao2019的对比，我们会有的
4. 因为ROC是不带标注的数据集，很多数据都是没有明显的风格偏向的
5. yao2019的数据之后放上

# #1

### 中心意见

1. 没有定义好Style，把Style的定义转到了类似的emotion上；导致最后的解决方法没有准确性

2. 认为我们的标注者一共只有三个人

3. 认为我们设计的标注提问有问题
- Flency Consistency Style这几个概念很混淆，不能很好地衡量输出的质量
- 标注的问题太抽象，导致有很多主观因素在里面

4. 通过比较进行评价往往能获得较高的得分

### 回复

1. - 79行我们已经定义了Style是根据word进行划分，而不是简单地说故事的风格就是情感（也就是不是根据不同情感划分故事为不同的类别，而是情感类故事是一种style）
- 有了基于词的定义之后，就能够定量分析我们生成的故事风格是什么
2. 不是一共三个人，是每一篇会让三个不同的人标注
3. - 对于标注者而言，Flency和Consistency确实有点混淆的意思，但是是一种衡量故事流畅和逻辑性的一个指标。（我们是为了保证在故事逻辑性流畅的前提下，能够有风格倾向）
- 因为所有的baseline和我们的system都是在同一个数据集上fine-tune的，不应该是数据的问题；而是模型（或者提出的方法）产生的效果
- which system is better应该是比较明确的问题

4. 确实是有这个问题的，无解

# #2

### 中心意见

1. 之前是否有过stylized story generation的文章
2. 对style的定义有些窄了，应该说本文注重词，但是也可以拓展到其他
3. 对keywords没有解码，而是直接作用于distribution，没搞懂
4. content-drive and style-driven
5. 一些语法错误

### 回复

# #3

### 中心意见

1. 模型缺乏一些创新
2. 消融实验
3. 有风格的故事占比太少
4. 需要和一些之前的故事生成模型进行比较

### 回复

1. 模型的一大创新是，没有生成keywords，而是使用了keywords distribution