# ACL ARR Oct STVD Rebuttal

## Response to Review \#1:

Thank you for your construstive comments and acknowledging the usefulness of our dataset. We carefully address each of your concerns as follow.

**Q1: What languages are covered in your dataset?**
**A:** We use English subtitle as the primary language for all the experiments. However, the dataset supports bilingual dialogues (English and Chinese) for future multimodal and multilingual extension.

**Q2: While the approach to annotate scene boundaries is clever, it is not very clear to me how annotators can accurately judge if a dialogue turn is the begining of a new scene.**
**A:** Thank you for your acknowledging our annotation approach. We consider the following aspects to narrow down the annotation ambiguities:
1. We utilize multimodal information in annotation, which can greatly facilitate the judgment of scene segmentation
2. We designed several rules as annotation constraints. For example, if current shot video with dialogue denotes a new environment, story or plot, the scene label of the dialogue utterance is annotated as `<start>`; if current dialogue utterance and interlocutors in corresponding short video start to talk about new thing, the topic label of the data item is annotated as `<start>`.
2. We provided detailed instructions and examples to every annotators before annotation. We show annotators 408 carefully annotated dialogue clips covering different situations as reference annotations.
3. We manually checked the annotation results and droped off bad cases of which error rate is more than 4% ; please refer to L260 for details.
<!-- For example, In Monica’s room, Rachel with her friends is talking about food, while topic turns, Joy begin to talk about the recent party, this change is a topic transition. After a while, the scene changes, Chandler and his new roommate start to chat in their room, the change here is scene transition. -->

We will update the above discussion in our revised submission.

**Q3: What do you mean by "... we modified the annotated boundaries as the end of a dialogue scene"? Does this mean that each scene ends when the dialogue ends?**
**A:** This is a format change, i.e. we only ask annotators to label the `<start>` tag of each scenes, and set the scene boundary before the next `<start>` tag.

**Q4: In Figure 2a, the y-axis doesn't look like a log scale.**
**A:** Thank you for your comments. We have changed the formats of y-axis in the revision.

## Response to Review #2:

Thank you for your construstive comments and acknowledging the contributions of our tassks. We carefully address each of your concerns as follow.

<!-- Q1: Even-though, the main challenges for topic and scene transitions are depicted in Figure 1, but it would be nice to more elaborate the reason why people usually care about this type of segmentation tasks, is it because they can be helpful in better understanding the dialogue and generating higher quality responses as it is shown in the end of the paper or they can have other applications and utilizations as well. -->
**Q1: It would be nice to more elaborate the reason why people usually care about this type of segmentation tasks**
**A:** We summarize the importance of segmentation task as follows:
- Scene segments play a key role in high level semantic understanding. Different from image-text pairs, video-grounded dialogues commonly suffer from frequent trendsends of high-dimentional visual and language information, making the machine unable to retrieve high-level semantics. Scene segments and topic segments, on the other hand, provide a feasible solution to facilitate the understanding process.
- Both scene transition and topic transition are very common in open-domain video-grounded dialogue and story-based long videos. However, most of currently popular works are QA-based datasets, e.g., Visual Dialog. Moreover, in our datasets and other open-domain multimodal datasets such as Twitch-Fifa, the semantic relation between video and dialogue are much weaker. Therefore, existing methods perform much worse on these datasets than QA-focused datasets. We believe the dialogue scene and topic information are the essential factors that contribute to such holistic understanding.
- Our segment transition is beneficial to spatio-temperol vision semantic understanding. As discussed in [1], such transitation can be leveraged to learn temporal relations among visual objects.

[1] Ji et. al., Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs, in CVPR 2020.

**Q2: Have authors done experiments to compare their proposed model's performance on other datasets such as MovieNet?**
**A:** Our models are specifically designed for scene&topic-aware video-grounded dialogue segments. Some key modules are missing and cannot be properly applied to other datasets, e.g. MovieNet contains no topic segment boundaries, and existing dialogue topic segment datasets are all text-only. Nevertheless, we believe previous success of model design on MovieNet or other datasets can also be easily migrated to our tasks with additional module design.

<!-- As authors also mentioned in the limitations part of the paper, low values of the automatic metrics for computing the response quality could be their inappropriateness for comparing the generated responses with only the one existing reference response. -->
**Q3: It is encouraged to conduct some experiments to ask human about the quality of the generated responses or even use model-based automatic metrics (analyzed in "A Comprehensive Assessment of Dialog Evaluation Metrics", Yeh et al 2021) that can be better applicable in such scenarios.**
**A:** Thank you for your suggestion. We are working on human evaluation. The results will be added with case study in the revision.

<!-- or even talk about their main differences with the proposed model to make it clear what are their differences. -->
**Q4: It would be nice to explain the compared baseline models for different tasks in the Related Work section (not in the appendix)**
**A:** Thank you for your advice. We will summarize add the main differences as below in the **Related Work**:

> - ShotCol adopt self-supervised contrastive learning method to learn a representation for shot
> - Bassal leverages pseudo-boundaries and several pretext tasks to learn a shot contextual representation
> - GreedySeg proposes a greedy algorithm to check if the interval is a dialogue topic segmentation.
> - Bert+CS construct a training corpus by contrastive learning strategy, based on which they design a BERT-based utterance-pair coherence scoring (CS) model for segment boundary prediction.

**Q5: Some minor grammatical/syntactical errors**
A: Apologize. We have proofread and fixed all the grammatical errors.

## Response to Review #3:

Thank you for your detailed reivew and comprehensive comments. We carefully address each of your concerns as follow. All the lacked details and analysis will be revealed in the revised submission.

<!-- , what kind of topics are there, how the error checking is done, and what kind of error is manually checked, how many data is dropped based on the manual checking process, what are the common problems during the annotation, etc. Since the main contribution is about proposing a new dataset & benchmark, it would be crucial for authors to elaborate a finer detail on the dataset collection and annotation -->
**Q1: The paper lacks of detail in the annotation process such as how a topic & topic boundary is defined.**
**A:** Thank you for your advice.
For **scene definition**, since both video scene segmentation and dialogue topic segmentation themselves are not newly proposed tasks, we follow previous work and define scene as in Line 199:
> a scene is a plot-based semantic unit in which a certain activity occurs among a specific group of individuals.

For **annotation procedure**, we made several rules for annotators. In summary, i) scene transition can be decided through the situation, i.e environment, costume props, dialogue topic and etc.; ii) dialogue topic transition is judged by if the interlocutors are talking about the same thing.
<!-- For dialogue topic genres, we are unable to give a precise definition for certain open-domain dialogue topic at current stage, we only label the topic boundaries. In the future, we will summarize some common dialogue topics. -->
We discuss our **error checking procedure** in **Section 3** Annotation Process:
> we drop off 4% bad cases (7.7K dialogues).
>
Unfortunately, most unqualified annotation are made by two irresponsible students. We will release a technical report about the annotation details.
<!-- We would also like to add any unclear details you want. -->

<!-- Based on the description, the annotation procedure and Figure 1, it seems like the data does not have a video segment per dialogue. -->
**Q2: The paper lacks of detail on how the alignment between dialogue turn and short video is done.**
**A:** In our setting, topic segmentations are fine-grained splits of scene segments w.r.t. dialogue topics. Different dialogue scene segments do not share semantic connections, which thereby indicates topic boundaries as well. We proposed **a multimodal annotation method** to ensure the pattern, where we show annotators both videos and subtitles to avoid the uncertainty of scene boundaries and topic boundary. For the situtation where dialogue topic transcends multiple scenes, we will ask the annotators to note these special cases and filter out them in the last. In other words, each end of a scene would come at the end of a dialogue topic. Because “dialogue scene is a plot-based semantic unit”. We will reserve these special cases where dialogue topic segment transcends multiple scenes for further research. In L240, we also show the statistic that each scene segment contains 1.88 topic segments.

**Q3: Human evaluation is not conducted for the response generation task, which is crucial for an open-domain dialogue system. Additionally, there is no case study of the generated examples vs the ground truth vs the baselines.**
**A:** Thank you for the kind advice. Indeed, we are in the process of performing human evaluation of our results on AMT. The results as well as case studies will be added to the revision.

<!-- While there are a few mentions of why scene segmentation could benefit video-grounded dialogue understanding,
That being said, outside of the intuitive example using the Memento movie reference, the paper doesn’t really show the importance of both of them to dialogue discourse modeling through hard evidence, quantifiables, or findings in existing works.
-->

**Q4: There is hardly any explanation why topic segmentation is important.**
**A:** Thank you for the comments. We summarize the importance of segmentation task as follows:
- Scene segments play a key role in high level semantic understanding. Different from image-text pairs, video-grounded dialogues commonly suffer from frequent trendsends of high-dimentional visual and language information, making the machine unable to retrieve high-level semantics. Scene segments and topic segments, on the other hand, provide a feasible solution to facilitate the understanding process.
- Both scene transition and topic transition are very common in open-domain video-grounded dialogue and story-based long videos. However, most of currently popular works are QA-based datasets, e.g., Visual Dialog. Moreover, in our datasets and other open-domain multimodal dataset such as Twitch-Fifa, the semantic relation between video and dialogue are much weaker. Therefore, existing methods perform much worse on these datasets than QA-focused datasets. We believe the dialogue scene and topic information are the essential factors that contribute to such holistic understanding.
- Our segment transition is beneficial to spatio-temperol vision semantic understanding. As discussed in [1], such transitation can be leveraged to learn temporal relations among visual objects.

[1] Ji et. al., Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs, in CVPR 2020.

<!-- In line 283 and 292, it is shown that a dialogue topic label corresponds to a certain dialogue clip (i.e., utterance), while a dialogue scene label corresponds to a certain video clip. -->
<!-- Also, if these multiple modalities are used to train SWST, “Video classification head” and “Text classification head” in Figure 4 seem a bit misleading. -->
**Q5: Why are they formulated this way while both dialogue and video are used to train SWST (Section 4.2 and Figure 4)?**
**A:** Our dataset is annotated with the help of multimodal information. Specifically, we adopted the currently popular single stream multimodal for our scene/topic segmentation task and use different classification head for different task. Please refer to L231 for details. We will highlight the above information in our revision.

**Q6: Why do you specifically use micro f1 for the scene segmentation task and macro f1 for the topic segmentation task?**
**A:** We compared all commonly used metrics on video scene understanding, including precision, recall. We find micro-F1 most suitable for scene segmentation because it is more tolerable to non-transition prediction.

**Q7: Could you please explain what “the dialogues in our dataset made by playwright…” in line 547 means?**
**A:** All the dialogues come from subtitles which are human generated conversations.

## Response to Review #4:
Thank you for your detailed reivew and kind suggestions. We carefully address each of your concerns as follow.

<!-- BERT is an interesting choice for text backbone. While it's better than, e.g., a BoW model, it's over 4 years old at this point ---
s, e.g., T5 (-Large/-3B/-11B)
-->

**Q1: Did the authors try (esp. for text only baselines) stronger text backbone?**
**A:** Our models can definately be replaced with more advanced backbones. Whereas, our key point is to demonstrate the effectiveness of our novel multimodal segment task. Unfortunately, due to the massive computational cost, we are unable to apply SOTA PLMs as our backbones. However, we are very delight to see further work which apply stronge backbones on our benckmark.

<!-- , as with the other tasks --- it's somewhat conspicuous that it's missing, but I might be misunderstanding. It also would have been nice to have more information about the "text-only" baselines. And while I appreciate their inclusion --- similar to my first concern, it would have been nice to have stronger text-only language classification/generation baselines -->
**Q2: It would have been nice to see a text only baseline for the scene segmentation task.**
**A:** Thank you for your suggestion. We performed the text-only experiments as reference. The results are shown in the below table.

| Model | mIoU | AP | F1 |
|-----------|----------|-----------|-------|
| SWST (text-only) | 0.37 | 0.36 | 0.33 |

We agree that stronger PLMs may demonstrate better performance in multimodal understandings, similar potentials have been shown in recent trends of prompt design. However, we believe the fusion and alignment of complex video and dialogue data is also an essential research theme that require further exploration. This research work, as stated in **Q1**, is to step towards fine-grained multimodal understanding rather than improve performance scores. Again, we will be delightful to see strong PLMs applied to our benchmark.

**Q3: It would have been nice to have a measure of human agreement/upper bound for the segmentation tasks. While the data validation and annotation procedures are discussed, an estimate of human performance on the task would be nice to know, and (at least for the classification tasks) seem plausible to acquire.**
**A:** We have followed your advice to measure the human agreement for taks: the upper bound for the scene segmentation task is 0.92; the upper bound for the topic segmentation task is 0.90.

**Q4: There are few error analyses or qualitative explorations conducted --- when are segmentations correct/incorrect? For next-utterance predictions, when does the model perform well/poorly?**
**A:** Good question. We have carefully investigated into the error analyses among different utterance lengths, utterance types. Currently, we didn't find a clear cue on the relations with these unique factors. Our hypothesis is that the segmentation task requires a comprehensive semantic understanding of multimodal signals. For next-utterance predictions, we observed that the vanilla model is easy to generate the replicated response or meaningless reply, such as "No, I don’t know", "Hi, ", especially when the context are greetings. We will add error analysis and case study in the revision.

**Q5: The generation result scores seem orders of magnitude smaller than most tasks -- are the results really this low?**
**A:** We will opensource all the code and evaluation scripts upon acceptance. We carefully checked the evaluation and confirmed the corretness of our results. Of note, similar magnitude can also be observed in other multimodal dialogue datasets, e.g. OpenVidial with Bleu4: 1.97.