# NSR ICLR23 Rebuttal
Reviews:
https://openreview.net/forum?id=m7rFrsO0YWb

## General comments for all reviewers and the summary of Revision

Dear all reviewers,

We are grateful for your constructive comments and helpful feedback. We are very encouraged by the acknowledgment that our paper "explores an important and growing domain," (Reviewer w6P6) and "is clearly written and well organized" (Reviewers w6P6, kopJ, oDdz), the proposed idea is intuitive and sound (Reviewers w6P6, j25h), the proposed model achieves strong generalization (all reviewers), and the related work is comprehensive and clear (Reviewers w6P6, oDdz).

To address your main concerns, we have done our best to improve our work and revise the paper accordingly. The revised texts in the draft are highlighted in blue. Please refer to the newest draft for the details. The revised parts are also summarized as follows:

1. We put the "Related Work" section right after "Introduction" and add a comparison to previous neural-symbolic methods. (Suggested by Reviewers w6P6 & kopJ)
2. We conduct a machine translation experiment using the English-French translation task from Lake & Baroni (2018), to explore how well the proposed NSR model handles real-world tasks. We put the details on this dataset and results in Appendix, Section A.4, due to the page limit. We also provide a discussion on the potential challenges when applying NSR to real-world tasks in the "Conclusion and Discussion" section. (Suggested by Reviewers w6P6 & j25h)
3. We add an ablation study to analyze the performance of individual components of NSR. We put the results and analysis of this ablation study in Appendix, Section A.5. (Suggested by Reviwer w6P6)
4. We add a comparison to a more recent baseline from Ontanón et al. (2022) in Table 1. (Suggested by Reviwers kopJ & oDdz)
5. We share our code at [bit.ly/nsr-iclr23](https://bit.ly/nsr-iclr23) and will host the code in a GitHub repository and share the trained models and experimental logs in the future.
6. some writing revision. (Suggested by Reviewer oDdz)

We appreciate all the suggestions made by reviewers to improve our work. It is our pleasure to hear your feedback and we look forward to answering your follow-up questions.

Paper264 Authors

## Reviewer w6P6
Thank you for the constructive feedback!

We very much appreciate your efforts in reviewing our paper and providing insightful feedback. It is encouraging for us to see your positive comments on the **Domain** ("important and growing"), **Experiments and results** ("good results across multiple tasks and compares to multiple suitable baselines", "The descriptions … are clear, well structured and easy to follow"), **Modeling section** ("The discussion is interesting and gives solid motivation"), and **Related work** ("does a good job contextualizing the approach …").

We address your concerns as follows:

> Related work comparison to specific recent approaches on the explored tasks

Our approach belongs to the third class ("Symbolic Scaffolding") discussed in Related Work; the most related method is the NeSS model proposed by Chen et al. (2020), which requires specific domain knowledge to design a symbolic stack machine with special operations (e.g., "CONCAT_M," "CONCAT_S"). Compared with NeSS, the proposed NSR model has two significant advantages:
(1) The dependency parser in NSR adopts a standard stack without requiring any special stack operation beyond the standard ones (Shift, Left-Arc, Right-Arc).
(2) The proposed deduction-abduction algorithm for learning NSR does not require a specialized curriculum for SCAN, as required by NeSS.

Thus, NSR has better transferability than NeSS, as reflected by the experimental results (Table 1): NSR achieves 100% accuracy on both SCAN and PCFG, while NeSS succeeds on SCAN but fails on PCFG. We will make the comparison clearer in the revised draft.

> Complicated approach and undifferentiable.

Being complicated and undifferentiable is a common weakness for most neural-symbolic approaches, including NeSS (Chen et al.2020), LANE (Liu et al. 2020), and the proposed NSR model, despite their better generalization than purely neural-based models like Transformer. We regard it **as an important future direction** to simplify these hybrid models and explore ways to make it differentiable.

On the other hand, when designing NSR, we tried our best to simplify it and make it applicable for more domains, i.e., the dependency parser and the program induction are adopted in the most general form, without adding any domain-specific design. Although still undifferentiable, the proposed deduction-abduction algorithm makes the training more straightforward. It does not require a manually-designed curriculum, compared with previous neural-symbolic models like NeSS.

> The model is evaluated on synthetic data only.

To explore how well the proposed NSR model is applicable to real-world tasks, We run a proof-of-concept machine translation experiment using the English-French translation task from Lake & Baroni (2018), which is also used by previous works (Li et al., 2019; Chen et al., 2020; Kim, 2021) to explore their methods on realistic domains. Compared to synthetic tasks like SCAN and PCFG, this translation task contains more complex and ambiguous rules. To evaluate compositional generalization, for a word "dax," the training data contains only one pattern of sentence pairs ("I am daxy," "je suis daxiste"), but test data contains other patterns (e.g., "you are not daxy," "tu n es pas daxiste").

We apply the proposed NSR model to this task. The results are as follows:

| Model | Accuracy |
|-------------------|:------------:|
|Seq2Seq (Lake & Baroni, 2018) | 12.5 |
|Primitive Substitution (Li et al., 2019)| 100.0 |
|NeSS (Chen et al., 2020) | 100.0 |
|NSR (ours) | 100.0 |

Similar to previous methods (Primitive Substitution and NeSS), NSR achieves a 100\% generalization accuracy on this task. This experiment shows that NSR has the promise to be applied to real-world tasks. We will add the above results, more details on this dataset, and the experimental setup into the revised draft.

> Suggestion: focusing on the reasoning modules

Thank you for the suggestion! In the HINT experiment, we have already conducted experiments using symbol inputs (as shown by the columns of "Symbol Input" in **Table 2**), in which the model only has the parsing and the reasoning modules, without the perceptual module. We will emphasize these results and highlight them in the revised draft.

> Evaluating each component independently

Since only the HINT benchmark provides intermediate results for evaluation, we evaluate the three components of NSR individually on HINT by offering ground truth to other modules. The results are as follows:

| Module | Accuracy |
|-------------------|:------------:|
|Neural Peception| 93.51 |
|Dependency Parsing| 88.10 |
|Program Induction| 98.47 |

Overall, each module achieves high accuracy. For Neural Perception, most errors come from the two parentheses, "(" and ")", because they are visually similar. For Dependency Parsing, we analyze the parsing accuracies for different concept groups: digits (100\%), operators (95.85\%), and parentheses (64.28\%). The parsing accuracy of parentheses is much lower than those of digits and operators. As long as digits and operators are correctly parsed in the parsing tree, where to attach the parentheses does not influence the final results because parentheses have no semantic meaning. For Program Induction, we can manually verify that the induced programs (Fig. 4) have correct semantics. The errors in the above table are caused by exceeding the recursion limit when calling the program for multiplication. We will add the above results and analysis in the revised draft.

We hope the above response can resolve your questions and concerns. Please let us know if there is any further question!

## Reviewer kopJ
Thank you for the constructive feedback!

We very much appreciate your efforts in reviewing our paper and providing insightful feedback. It is encouraging for us to see your positive comments that "the proposed idea is sound and shows superior performance," "the paper is clearly written and well organized," and "the proposed deduction-abduction algorithm is sound and design choices are also well-explained."

We address your concerns as follows:

> Lack of comparison versus more recent baselines such as Ontanón et al. (2022)

Thanks for pointing it out! Ontanón et al. (2022) evaluate the compositional generalization of Transformer variants and conduct experiments on SCAN and PCFG. We cite the best results from Ontanón et al. (2022) and compare them with the proposed NSR model as follows:
|Model |SCAN-Jump |SCAN-Length |PCFG-Systematicity |PCFG-Productivity |
|-------------------|:------------:|:------------:|:------------:|:------------:|
| Transformer (Ontanón et al., 2022) | 0.0 | 19.6 | 82.8 | 63.4 |
| NSR (ours) | 100.0 | 100.0 | 100.0 | 100.0 |

From the above results, we can see that NSR demonstrates much stronger generalization than the state-of-the-art Transformer. We will add the above comparison in the revised draft.

> Contrast NSR against prior work in the "Related Work" section.

Thank you for the suggestion! Our approach belongs to the third class ("Symbolic Scaffolding") discussed in Related Work; the most related method is the NeSS model proposed by Chen et al. (2020), which requires specific domain knowledge to design a symbolic stack machine with special operations (e.g., "CONCAT_M," "CONCAT_S"). Compared with NeSS, the proposed NSR model has two significant advantages:
(1) The dependency parser in NSR adopts a standard stack without requiring any special stack operation beyond the standard ones (Shift, Left-Arc, Right-Arc).
(2) The proposed deduction-abduction algorithm for learning NSR does not require a specialized curriculum for SCAN, as required by NeSS.
Thus, NSR has better transferability than NeSS, as reflected by the experimental results (Table 1): NSR achieves 100% accuracy on both SCAN and PCFG, while NeSS succeeds on SCAN but fails on PCFG. We will make the comparison clearer and move the related work section to an earlier point in the revised draft.

We hope the above response can resolve your questions and concerns. Please let us know if there is any further question!

## Reviewer j25h
Thank you for the constructive feedback!

We very much appreciate your efforts in reviewing our paper and providing insightful feedback. It is encouraging for us to see your positive comments that "NSR achieve(s) impressive generalization," "the three-module design is intuitive and general," and "NSR has good interpretability."

We address your concerns as follows:

> The NSR is only tested on synthetic tasks. The limitation of applying NSR in real-world tasks is not discussed either.

To explore how well the proposed NSR model is applicable to real-world tasks, we run a proof-of-concept machine translation experiment using the English-French translation task from Lake & Baroni (2018), which is also used by previous works (Li et al., 2019; Chen et al., 2020; Kim, 2021) to explore their methods on realistic domains. Compared to synthetic tasks like SCAN and PCFG, this translation task contains more complex and ambiguous rules. To evaluate compositional generalization, for a word "dax," the training data contains only one pattern of sentence pairs ("I am daxy," "je suis daxiste"), but test data contains other patterns (e.g., "you are not daxy," "tu n es pas daxiste").

We apply the proposed NSR model to this task. The results are as follows:

| Model | Accuracy |
|-------------------|:------------:|
|Seq2Seq (Lake & Baroni, 2018) | 12.5 |
|Primitive Substitution (Li et al., 2019)| 100 |
|NeSS (Chen et al., 2020) | 100 |
|NSR (ours) | 100 |

Similar to previous methods (Primitive Substitution and NeSS), NSR achieves a 100\% generalization accuracy on this task. This experiment shows that NSR has the promise to be applied to real-world tasks. Despite the perfect accuracy in this proof-of-concept experiment, we indeed anticipate potential challenges of applying NSR to real-world tasks: (1) The noisy and numerous concepts in real-world tasks have a large space of grounded symbol system and might slow the training of NSR; (2) The functional programs in NSR are deterministic and thus not able to represent probabilistic semantics in real-world tasks, e.g., in machine translation, there might be multiple ways to translate a single sentence.

We will add the above results, more details on this dataset, and the experimental setup into the revised draft.

> The paper is difficult to follow, important diagrams and figures are put in the appendix. This also makes the model difficult to reproduce.

Thanks for this valuable comment. We strive to make our paper easy to follow. Due to the page limit, we have to focus on the core idea and the intuition behind the model design in the main text, while putting **task-specific details** in the appendix. This way of organizing the paper is commented by other reviewers as "clear, well-structured and easy to follow", "clearly written and well organized", and "clearly described."

We would love to further improve the clarity of our paper if you have a more specific suggestion on which part should move to the main text to make the method easier to follow. We are very happy to discuss and follow your suggestions!

Regarding reproducibility, our code can be found at [bit.ly/nsr-iclr23](https://bit.ly/nsr-iclr23) and we will host the code in a GitHub repository and share the trained models and experimental logs in the future.

We hope the above response can resolve your questions and concerns. Please let us know if there is any further question!

## Reviewer oDdz

Thank you for the constructive feedback!

We very much appreciate your efforts in reviewing our paper and providing insightful feedback. It is encouraging for us to see your positive comments that "NSR demonstrates a good degree of generalization," "the related work is fairly comprehensive in topic, and clarity," "Fig 3 is fairly convincing."

We address your concerns as follows:

> More recent alternatives to Chen & Manning (2014) for dependency parsing, including Mrini et al (2020).

Thank you for the suggestion! When designing the individual modules for NSR, we intend to keep them as simple as possible and choose Chen & Manning (2014) for dependency parsing, which is powerful enough for the studied tasks in our experiments. Of note, NSR by design does not rely on the structure in Chen & Manning (2014); instead, it is compatible with more advanced alternatives for parsing like Mrini et al (2020), which is potentially helpful for applying NSR to domains with more complex syntax.

> It would be preferable to explore the space of program induction to a greater degree

Thank you for the suggestion! We will enrich the discussion of program induction in the revised draft as follows:
"
Program induction, i.e., synthesizing programs from input-output examples,
was one of the oldest theoretical frameworks for concept learning within artificial intelligence (Solomonoff, 1964). Recent advances in program induction focus on training neural networks to guide the program search (Kulkarni et al., 2015; Lake et al., 2015; Balog et al., 2017; Devlin et al., 2017; Ellis et al., 2018a;b). For example, Balog et al. (2017) trained a neural network to first predict the program's properties given the input-output pairs, and then use the neural network's predictions to augment search techniques from the programming languages community. Recently, Ellis et al. (2021) devised a neural-guided program induction system, DreamCoder, which can efficiently discover interpretable, reusable, and generalizable programs across a wide range of domains, including both classic inductive programming tasks and creative tasks such as drawing pictures and building scenes. DreamCoder adopts a wake-sleep Bayesian learning algorithm to extend the program space with new symbolic abstractions and train the neural network on imagined and replayed problems.
"

> Compare NSR against the approach in Ontañón et al (2022)

Thanks for pointing it out! Ontanón et al. (2022) evaluate the compositional generalization of Transformer variants and conduct experiments on SCAN and PCFG. We cite the best results from Ontanón et al. (2022) and compare them with the proposed NSR model as follows:
|Model |SCAN-Jump |SCAN-Length |PCFG-Systematicity |PCFG-Productivity |
|:-------------------:|:------------:|:------------:|:------------:|:------------:|
| Transformer (Ontanón et al., 2022) | 0.0 | 19.6 | 82.8 | 63.4 |
| NSR (ours) | 100.0 | 100.0 | 100.0 | 100.0 |

From the above results, we can see that NSR demonstrates much stronger generalization than the state-of-the-art Transformer. We will add the above comparison in the revised draft.

> Minor points.

Thank you for the valuable suggestions! We will fix the mentioned issues in the revised draft.

We hope the above response can resolve your questions and concerns. Please let us know if there is any further question!