# Responses to Reviewer 2fke (R2)
Thanks for your response and acknowledge on the quality of the paper!
Again, We greatly appreciate your valuable time and input!
# Responses to Reviewer czPr (R2)
Thank you for your additional feedback and continued engagement! We are glad that our initial responses adequately answered most of your questions. We would like to take the opportunity to further address your remaining question. Below, please find our detailed responses to your questions.
---
> ***Q1:*** "This code-gen results look promising! Do you plan to include them into your paper? As I think the current writing of the paper is mainly focusing on drug discovery task itself? If so, can you elaborate a bit which part of your algorithm design is general, and which part is specifically for drug discover problem?"
***A:*** Thanks for raising the question. Yes, we will include code-gen experiments into our paper. The primary contribution of this paper is methodological, and our approach can be applied to a broad spectrum of sequential generation problems.
We focus on the drug discovery problem in this paper because we believe it represents an important and challenging issue to address. Compared to the simple pass/fail outcomes in some tasks, drug discovery presents a broader range of objectives to achieve. Therefore, the ERP algorithm with its components \{PH-UCT, e-steps forwards, Top-PK \} are all designed to be general; To address a new sequential generation problem, we only need to define an appropriate reward function and adapt a pre-trained LLM model to the new dataset.
> ***Q2:*** "... for the table of same computational budget, I still prefer counting the number of token and provide a table / figure where x axis is num_token and y-axis is performance, as different search strategy can have different pruning mechanism so the same num_rollout does not mean they use the same search budget."
***A:*** We apologize for a typo in the previous answer to "Q2". In fact, we have already conducted the experiment you requested, using the same budget as "counting the number of tokens". The error occurred in the X-axis of the table, where we mistakenly labeled it "Number of rollouts" instead of "Number of sampled tokens". Please find the corrected version of our answer below:
>> ***Q2:*** "I personally would like to see the comparison within the same computational budget ... number of token LLM output / input, which is related to real cost~... current comparison of number of rollouts can be misleading as some algorithm explore the space less..."
>>***A:*** Thank you for raising the question! In our evaluation of the main paper, we maintained the same budget for rollouts as referenced in PG-TD [1]. To address your concerns, we conducted experiments using an identical computational budget in terms of the number of tokens output/input by the LLM across ***7 budget constraints comparisons***, ranging from 1,024 (=$2^{10}$) to 65,536 (=$2^{16}$) tokens. The results in the following table clearly demonstrate that ERP consistently outperforms PG-TD under various token constraint conditions. This finding is in line with the results presented in the main paper. Moreover, we will update paper and the codebase to facilitate the reproduction of these experiments.
### Table : Average normalized reward
| <font color='#EE6363'>**Number of sampled tokens**</font> | ERP with e-etep=6 | PG-TD |
|--------------------|-------------------|-------|
| 1024 | 2.36 | 2.36 |
| 2048 | **2.28** | 2.25 |
| 4096 | **2.34** | 2.28 |
| 8192 | **2.39** | 2.35 |
| 16384 | **2.54** | 2.43 |
| 32768 | **2.60** | 2.57 |
| 65536 | **2.60** | 2.57 |
---
We hope our responses have addressed your remaining questions. Again, we truly appreciate of your valuable time and input! If you have any further inquiries, please kindly let us know. Thank you!
# Responses to Reviewer czPr (R1)
Thank you for the feedback on our work! Below please find our detailed responses to your questions.
---
> ***Q1:*** "I think the methodology-wise contribution in this paper is some tree search component (in terms of exploration and node selection), if it's that, probably it's better to also evaluate the method on general tree search benchmarks ... like coding..."
***A:*** Thank you for raising this question! Our paper's contribution is primarily methodological, allowing for its application across various domains, including code generation. To address your concern, we evaluated our method by comparing it to the existing state-of-the-art in code generation tasks, specifically PG-TD [1], using two fine-tuned language models {GPT-2, GPT-Neo} as provided by the work of PG-TD. This evaluation included three benchmarks from the APPS dataset [2] {APPS Intro, APPS Inter, APPS Comp} and one benchmark from the CodeContests dataset [3], assessed on two performance metrics: {pass rate, strict accuracy} and two tasks {Top N, Pass@K}. The pass rate represents the average percentage of private test cases that the generated programs successfully pass across all problems. Strict accuracy measures the percentage of problems for which the generated programs pass all the private test cases.
Due to limited time and computational resources, we confined the use of both PG-TD and ERP methods to a rollout of 64 and further configured the ERP with an e-step of 2. We reused the GPT-x models from the PG-TD release versions.
Our approach outperformed the current leading method, PG-TD, in all four code generation tasks across two LLM models, GPT-2 and GPT-Neo, on two metrics: pass rate and strict accuracy. We report Top N and Pass@K results, with N and K values of 15 and 20. Our results demonstrate that the ERP method consistently surpasses the PG-TD method for every selected N and K value. Finally, we will update both papers and the codebase to facilitate the reproduction of these additional experimental results.
### Table : Top n
| TOP N | Base Model | Algorithm | Pass Rate (%) - APPS Intro. | Pass Rate (%) - APPS Inter. | Pass Rate (%) - APPS comp. | Pass Rate (%) - CodeContests | Strict Accuracy (%) - APPS Intro. | Strict Accuracy (%) - APPS Inter. | Strict Accuracy (%) - APPS comp. | Strict Accuracy (%) - CodeContests |
|-------|------------|-----------|-----------------------------|-----------------------------|----------------------------|------------------------------|-----------------------------------|-----------------------------------|----------------------------------|------------------------------------|
| 15 | GPT-2 | PG-TD | 28.12 | 23.46 | 32.82 | 17.17 | 2.30 | 8.47 | 16.40 | 2.42 |
| - | - | ERP (Ours) | **30.66** | **24.53** | **34.45** | **18.83** | **2.50** | **8.93** | **17.00** | **3.03** |
| 15 | GPT-Neo | PG-TD | 29.16 | 24.51 | 35.60 | 15.84 | 2.40 | 9.03 | 19.10 | 2.42 |
| - | - | ERP (Ours) | **31.62** | **25.77** | **37.27** | **18.40** | **2.90** | **9.20** | **20.30** | 2.42 |
| 20 | GPT-2 | PG-TD | 28.18 | 23.93 | 33.42 | 17.18 | 2.30 | 8.67 | 16.80 | 2.42 |
| - | - | ERP (Ours) | **30.74** | **25.38** | **35.37** | **18.83** | **2.50** | **9.33** | **17.50** | **3.03** |
| 20 | GPT-Neo | PG-TD | 29.20 | 24.98 | 36.01 | 15.85 | 2.40 | 9.23 | 19.30 | 2.42 |
| - | - | ERP (Ours) | **31.74** | **26.61** | **38.31** | **18.40** | **2.90** | **9.57** | **20.80** | 2.42 |
### Table : Pass@K (Pass@15 and Pass@20)
| K for Pass@K | models | methods | Pass Rate (%) - APPS Intro. | Pass Rate (%) - APPS Inter. | Pass Rate (%) - APPS comp. | Pass Rate (%) - CodeContests | Strict Accuracy (%) - APPS Intro. | Strict Accuracy (%) - APPS Inter. | Strict Accuracy (%) - APPS comp. | Strict Accuracy (%) - CodeContests |
|--------------|--------|---------|-----------------------------|-----------------------------|----------------------------|------------------------------|-----------------------------------|-----------------------------------|----------------------------------|------------------------------------|
| K=15 | APPS GPT-2 | PG-TD | 27.87 | 23.23 | 32.73 | **16.87** | 2.20 | 8.37 | **16.60** | 2.42 |
| K=15 | - | ERP estep=2 | **28.42** | **23.48** | **32.86** | 16.61 | **2.30** | **8.57** | 15.90 | 2.42 |
| K=15 | APPS GPT-Neo | PG-TD | 28.93 | 24.27 | 35.56 | 15.57 | 2.40 | **9.07** | 19.00 | 2.42 |
| K=15 | - | ERP estep=2 | **29.30** | **24.42** | **35.90** | **16.86** | **2.50** | 8.90 | **19.70** | 2.42 |
| K=20 | APPS GPT-2 | PG-TD | 28.14 | 23.98 | 33.42 | 17.14 | 2.30 | 8.70 | 16.80 | 2.42 |
| K=20 | - | ERP estep=2 | **29.71** | **24.97** | **34.81** | **17.72** | **2.40** | **9.20** | **17.00** | 2.42 |
| K=20 | APPS GPT-Neo | PG-TD | 29.21 | 25.00 | 36.05 | 15.85 | 2.40 | 9.23 | 19.30 | 2.42 |
| K=20 | - | ERP estep=2 | **30.92** | **26.00** | **37.66** | **18.03** | **2.70** | **9.43** | **20.40** | 2.42 |
> ***Q2:*** "I personally would like to see the comparison within the same computational budget ... number of token LLM output / input, which is related to real cost~... current comparison of number of rollouts can be misleading as some algorithm explore the space less..."
***A:*** Thank you for raising the question! In our evaluation of the main paper, we maintained the same budget for rollouts as referenced in PG-TD [1]. To address your concerns, we conducted experiments using an identical computational budget in terms of the number of tokens output/input by the LLM across 13 budget constraints comparisons, ranging from 1, 024 (=2^10) to 65, 536 (=2^16) tokens. The results in the following table clearly demonstrate that ERP consistently outperforms PG-TD under various token constraint conditions. This finding is in line with the results presented in the main paper. Lastly, we will update paper and the codebase to facilitate the reproduction of these experiments.
### Table : Average normalized reward
| Number of rollouts | ERP with e-etep=6 | PG-TD |
|--------------------|-------------------|-------|
| 1024 | 2.36 | 2.36 |
| 2048 | **2.28** | 2.25 |
| 4096 | **2.34** | 2.28 |
| 8192 | **2.39** | 2.35 |
| 16384 | **2.54** | 2.43 |
| 32768 | **2.60** | 2.57 |
| 65536 | **2.60** | 2.57 |
> ***Q3:*** "... it's important to add Pass@K without tree search baselines (e.g., simply sampling LLM-IO multiple times, and select the one with highest rewards)..."
***A:*** Across all methods in Table 1, including the baselines, the ratio of valid molecules to all generated molecules are all close to 1.0. Thus the main improvement our work focuses on is not generating a molecule which is a valid molecule, but to efficiently generate many distinct molecules with desirable pharmaceutical properties for further discovery and engineering. Thus the notion of "pass" and the metric pass@k in coding does not directly apply. On the other hand, sampling LM multiple times without tree search is an important baseline, and such results of sampling from the LM multiple times (256 in the case of Table 1) and selecting the one with the highest rewards are included in the column of _Best Norm Reward_ and the row of _Sampling_ in Table 1. The results show that ERP outperforms the baselines in finding the best molecule, as well as more disctinct valid molecules.
> ***Q4:*** "seems that I didn't see MCTS as a baseline?"
***A:*** Thank you for pointing this out. Previously, one of our baselines is UCT (Upper Confidence bounds applied to Trees), which integrates upper confidence bounds with Monte Carlo Tree Search (MCTS), and uses Language Model (LM) in both expansion (Top-P and Top-K filtering) and evaluation (LM’s prompted generation) stages of the tree search algorithm.
Based on your advice, we introduced two more baselines to help illustrate the intrinsic complexity of this particular generation problem. Here we consider UCT, an instantiation of MCTS, with less or completely no use of the neural LM. The results are in the table below, which will be included in the next version of our paper.
There are two incrementally weakened algorithms. The weakest one is (b) UCT using LM in neither expansion nor evaluation, thus during expansion all tokens from the vocabulary are added and are considered equally likely. The experiment shows that no valid molecules were found with the rollouts of 512. The other finer-grained case is that (c) UCT uses LM in expansion but not in evaluation, so during the expansion stage the distribution and Top-k and Top-p filtering is used, and the more probable (according to the LM) continuation is prioritized for node selection if the scores are the same. Yet during evaluation, uniformly random sampling instead of a pre-trained LM’s prompted decoding is used. The performance of this baseline finds very few valid molecules. The table below shows experiments on 3CLPro, corresponding to the upper block of Table 1 in the reviewd paper.
### Table : Extended baselines
| | Average normalized reward | Ratio of valid molecules | Number of unique valid molecules found |
|-|-|-|-|
| UCT without LM (b) | Null | 0.0 | 0 |
| UCT using LM in expansion only (c) | 1.99 | 0.0001 | 1 |
| Beam search with pre-trained LM (as in Table 1) | 2.38 | 1.0 | 16 |
| Sampling with pre-trained LM (as in Table 1) | 2.39 | 1.0 | 1609 |
| UCT (as in Table 1) | 2.25 | 1.0 | 3013 |
| PG-TD (as in Table 1) | 2.59 | 1.0 | 2549 |
| ERP (as in Table 1) | 2.62 | 1.0 | 2575 |
> ***Q5:*** What if in some setup that we don't have access to oracle reward, can this method still work?
***A:*** Thank you for raising the question. Without oracle reward, then we will miss the Q(s, a) value in our PH-UCT function (suppose we set Q(s, a) to 0). However, one of the novelties of ERP, as mentioned by reviewer "jtjr" is the use of the e-step forward pass to evaluate branches even before rolling them out to determine their value. This approach reduces the need to fully explore all branches, thereby saving computational costs. Therefore, even in setups where we do not have access to an oracle reward, ERP can still achieve a balance between exploration and exploitation by reducing global uncertainty.
----
We hope these responses and additional experiments have adequately addressed your concerns. We appreciate your constructive feedback and look forward to further discussions!
[1] Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. "Planning with Large Language Models for Code Generation." In The Eleventh International Conference on Learning Representations. 2022.
[2] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. NeurIPS, 2021
[3] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R´emi Leblond, Tom Eccles, James Keeling, Felix Gimeno, and Agustin Dal Lago. Competition-Level Code Generation with AlphaCode. arXiv preprint arXiv:2203.07814, 2022.
# Responses to Reviewer TKSf （R1）
Thank you for the feedback on our work! Below please find our detailed responses to your questions.
---
> ***Q1:*** "The technical contributions are sound but limited. The authors propose to use the MCTS planning algorithm with multi-rewards to guide the transformer decoder for molecule generation. However, a tree search-based planning algorithm with LLM is not novel and has been proposed by many prior works. The reviewer thinks this part cannot provide new insights."
***A:*** Thank you for raising the concern. We acknowledge that tree search-based planning algorithms, when combined with Large Language Models (LLMs), have been utilized in solving problems such as code generation. The principal contribution of this paper is the introduction of the Entropy-Reinforced Planning (ERP) algorithm, which includes PH-UCT, E-steps, and Top-PK as follows:
PH-UCT: This algorithm adopts a strategy similar to UCB, selecting states and actions with high uncertainty or those that have never been visited before. This approach enables the agent to discover new molecule sequences that might not be accessible through stochastic search methods.
The E-steps represent another innovation, utilizing the e-step forward pass to evaluate branches even before they are fully expanded to ascertain their value. This strategy reduces the need to exhaustively explore all branches, thereby conserving computational resources.
Furthermore, we introduce Top-PK, a method that merges the benefits of both Top-P and Top-K to enhance the likelihood of selecting the next potential tokens that will lead to a valid high reward molecule.
In summary, the integration of these three components—PH-UCT, E-steps, and Top-PK—improves exploration and exploitation, aiding in the discovery of hidden spaces that yield high rewards.
> ***Q2:*** "The authors mentioned that relying solely on LLMs' decoding often results in the generation of invalid molecules due to a single misused token. However, this only happens when we represent each molecule as a state s in the context of SMILES and utilize search. Actually, we can achieve 100% validity by representing the molecule as sequential motifs."
***A:*** We acknowledge that representing each molecule as a sequence of motifs could improve validity when the tokens linking different motifs are correctly generated. However, relying solely on the decoding capabilities of Large Language Models (LLMs) still tends to maximize the probability of matching tokens (or motifs), which may lead to unbalanced exploration and exploitation. Therefore, the molecules generated in this manner may not be optimized for the desired objective functions.
> ***Q3:*** "Although the authors have provided the code for reproducing the paper, the implementation and dataset details are missing."
***A:*** Thank you for pointing out the missing details. Here are more details on the implementation and dataset as follows:
***Dataset detail.*** Pretrained LLM is trained using about 10.7 million druglike and in-stock molecules from ZINC20 [1] with standard reactivity. These molecules have a minimum sequence length of 8, an average length of 46, and a maximum length of 252. The biased LLMs are finetuned from the pretrained LLM using the cancer and covid dataset of Liu et al. 2023[2]. More specifically, 1 million compounds docked to 3CLPro~(PDB ID: 7BQY) protein associated with SARS-CoV-2 and the RTCB (PDB ID: 4DWQ) human cancer protein dataset with docking score in range [-14, -9], indicating string interactions, are selected for finetuning. When assessing the performance of our model, we go beyond solely checking if it can generate valid SMILES strings. Unlike code generation tasks, which are typically evaluated on a simple pass/fail basis depending on whether the code runs correctly, our evaluation for SMILES generation encompasses several additional pharmaceutical objectives: druglikeliness, docking score, synthesizability and solubility. This comprehensive approach ensures not only the generation of chemically valid structures but also evaluates their potential as effective drug candidates based on multiple critical factors.
***Implementation detail.*** To ensure a fair evaluation, we compare different algorithms based on similar setups. Specifically, in every comparison across all tables and charts, the algorithms share essential hyperparameters, such as the number of rollouts, along with the generation and prediction settings of the Transformer-based language models. Our hyperparameter tuning was constrained, focusing on a range of potential values as outlined in the following Tables. We then applied the same hyperparameter settings to assess the performance of different algorithms.
| Hyperparameter | Experimented values |
|----------------|---------------------|
| **GPT2 Pretraining and Fine-tuning** | - |
| Learning rate | $5 \times 10^{-5}$ |
| Batch size per GPU | 8 |
| # of GPU used | 8 |
| # of epochs | 10 |
| **GPT2 Fine-tuning** | - |
| # of epochs | 40 |
| **Tree Search with ERP** | - |
| Number of rollouts \(N\) | {32, 64, 128, 256, 512} |
| Exploration parameter \(c_p\) | {1, 4, 8} |
| LM top-p filter for expansion \(p\) | {0.9, 0.95} |
| LM top-k filter for expansion \(k\) | {15, 20} |
| LM beam size for evaluation \(b\) | {8, 16} |
| Forward step \(e\) | {1, 2, 3, 4, 5, 6} |
We trained our docking surrogate models using four nodes of the supercomputer where each node contains one 64-core CPU and four A100 GPU nodes. The training time for each model was approximately three hours. We conducted Transformer inference on a cluster that includes CPU nodes and GPU nodes (approximately two Nvidia GPUs). Based on the computing infrastructure, we obtained the wall-time comparison in Table as follows.
| Methods | Total Run Time |
|---------|----------------|
| **Pretrain GPT** | 9 hours |
| **Biased GPT** | 17 hours |
| **RL finetuned GPT** | 17 hours |
| **Sampling** | 10 min. |
| **PG-TD** | 30 min. |
| **ERP** | 40 min. |
We will update the above dataset and implementation details into the camera-ready version.
---
We hope our responses and additional experiments have addressed your concerns. If you have any further inquiries, please kindly let us know. Thank you!
# Responses to Reviewer jtjr (R1)
We appreciate your in-depth review of our work! Below please find our detailed responses to your questions.
---
> ***Q1:*** "The authors evaluated two benchmarks, on which they demonstrate ERP to be the SOTA. Further description of the datasets will be good to ascertain the complexity, number of sequence and types of desirable traits the dataset calls for to determine if 2 or more datasets are needed to validate ERP as SOTA."
***A:*** Thanks for raising the question! Pretrained LLM is trained using about 10.7 million druglike and in-stock molecules from ZINC20 [1] with standard reactivity. These molecules have a minimum sequence length of 8, an average length of 46, and a maximum length of 252. The biased LLMs are finetuned from the pretrained LLM using the cancer and covid dataset of Liu et al., 2023 [2]. More specifically, 1 million compounds docked to 3CLPro~(PDB ID: 7BQY) protein associated with SARS-CoV-2 and the RTCB (PDB ID: 4DWQ) human cancer protein dataset with docking score in range [-14, -9], indicating string interactions, are selected for finetuning.
When assessing the performance of our model, we go beyond solely checking if it can generate valid SMILES strings. Unlike code generation tasks, which are typically evaluated on a simple pass/fail basis depending on whether the code runs correctly, our evaluation for SMILES generation encompasses several additional pharmaceutical objectives: druglikeliness, docking score, synthesizability and solubility. This comprehensive approach ensures not only the generation of chemically valid structures but also evaluates their potential as effective drug candidates based on multiple critical factors.
Finally, To further address your concern and demonstrate the effectiveness of the ERP algorithm, we evaluated our method by comparing it to the existing state-of-the-art in code generation tasks, specifically PG-TD, using two base models, {GPT-2, GPT-Neo}. This evaluation included three benchmarks from the APPS dataset [4] {APPS Intro, APPS Inter, APPS Comp} and one benchmark from the CodeContests dataset [5], assessed on two performance metrics: {pass rate, strict accuracy} and two tasks: {Top N, Pass@K}. The pass rate represents the average percentage of private test cases that the generated programs successfully pass across all problems. Strict accuracy measures the percentage of problems for which the generated programs pass all the private test cases.
Due to limited time and computational resources, we confined the use of both PG-TD and ERP methods to a rollout of 64 and further configured the ERP with an e-step of 2. We reused the GPT-x models from the PG-TD release versions.
The results are demonstrated in following tables. Our approach outperformed the current leading method, PG-TD, in all four code generation tasks across two LLM models, GPT-2 and GPT-Neo, on two metrics: pass rate and strict accuracy. We report Top N and Pass@K results, with N and K values of 15 and 20. Our results demonstrate that the ERP method consistently surpasses the PG-TD method for every selected N and K value. Finally, we will update both papers and the codebase to facilitate the reproduction of these additional experimental results.
### Table : Top n
| TOP N | Base Model | Algorithm | Pass Rate (%) - APPS Intro. | Pass Rate (%) - APPS Inter. | Pass Rate (%) - APPS comp. | Pass Rate (%) - CodeContests | Strict Accuracy (%) - APPS Intro. | Strict Accuracy (%) - APPS Inter. | Strict Accuracy (%) - APPS comp. | Strict Accuracy (%) - CodeContests |
|-------|------------|-----------|-----------------------------|-----------------------------|----------------------------|------------------------------|-----------------------------------|-----------------------------------|----------------------------------|------------------------------------|
| 15 | GPT-2 | PG-TD | 28.12 | 23.46 | 32.82 | 17.17 | 2.30 | 8.47 | 16.40 | 2.42 |
| - | - | ERP (Ours) | **30.66** | **24.53** | **34.45** | **18.83** | **2.50** | **8.93** | **17.00** | **3.03** |
| 15 | GPT-Neo | PG-TD | 29.16 | 24.51 | 35.60 | 15.84 | 2.40 | 9.03 | 19.10 | 2.42 |
| - | - | ERP (Ours) | **31.62** | **25.77** | **37.27** | **18.40** | **2.90** | **9.20** | **20.30** | 2.42 |
| 20 | GPT-2 | PG-TD | 28.18 | 23.93 | 33.42 | 17.18 | 2.30 | 8.67 | 16.80 | 2.42 |
| - | - | ERP (Ours) | **30.74** | **25.38** | **35.37** | **18.83** | **2.50** | **9.33** | **17.50** | **3.03** |
| 20 | GPT-Neo | PG-TD | 29.20 | 24.98 | 36.01 | 15.85 | 2.40 | 9.23 | 19.30 | 2.42 |
| - | - | ERP (Ours) | **31.74** | **26.61** | **38.31** | **18.40** | **2.90** | **9.57** | **20.80** | 2.42 |
### Table : Pass@K (Pass@15 and Pass@20)
| K for Pass@K | models | methods | Pass Rate (%) - APPS Intro. | Pass Rate (%) - APPS Inter. | Pass Rate (%) - APPS comp. | Pass Rate (%) - CodeContests | Strict Accuracy (%) - APPS Intro. | Strict Accuracy (%) - APPS Inter. | Strict Accuracy (%) - APPS comp. | Strict Accuracy (%) - CodeContests |
|--------------|--------|---------|-----------------------------|-----------------------------|----------------------------|------------------------------|-----------------------------------|-----------------------------------|----------------------------------|------------------------------------|
| K=15 | APPS GPT-2 | PG-TD | 27.87 | 23.23 | 32.73 | **16.87** | 2.20 | 8.37 | **16.60** | 2.42 |
| K=15 | - | ERP estep=2 | **28.42** | **23.48** | **32.86** | 16.61 | **2.30** | **8.57** | 15.90 | 2.42 |
| K=15 | APPS GPT-Neo | PG-TD | 28.93 | 24.27 | 35.56 | 15.57 | 2.40 | **9.07** | 19.00 | 2.42 |
| K=15 | - | ERP estep=2 | **29.30** | **24.42** | **35.90** | **16.86** | **2.50** | 8.90 | **19.70** | 2.42 |
| K=20 | APPS GPT-2 | PG-TD | 28.14 | 23.98 | 33.42 | 17.14 | 2.30 | 8.70 | 16.80 | 2.42 |
| K=20 | - | ERP estep=2 | **29.71** | **24.97** | **34.81** | **17.72** | **2.40** | **9.20** | **17.00** | 2.42 |
| K=20 | APPS GPT-Neo | PG-TD | 29.21 | 25.00 | 36.05 | 15.85 | 2.40 | 9.23 | 19.30 | 2.42 |
| K=20 | - | ERP estep=2 | **30.92** | **26.00** | **37.66** | **18.03** | **2.70** | **9.43** | **20.40** | 2.42 |
> ***Q2:*** "A deeper analysis on why or how PH-UCT works to improve the performance... For instance, how is ERP balancing exploration-exploitation? ..."
***A:*** Thank you for the question! ERP balances exploration and exploitation through its three components: PH-UCT, e-steps forward guidance, and TOP-PK. These components play roles in different stages. PH-UCT and e-steps are utilized during the selection stage of tree expansion, enhancing the bonus term to include the probability of the next token as determined by the Transformer, as well as the entropy of its subsequent e-steps. This approach encourages the tree search to select tokens that balance between exploration and exploitation, while reducing global uncertainty. On the other hand, TOP-PK combines the advantages of both TOP-P and TOP-K methods, which serves at the expansion stage. Unlike standard MCTS, which may randomly sample a token, often leading to an invalid molecule, TOP-PK constrains tree expansion to the most probable next tokens within a limited rollout cost. These components tend to 1) limit excessive exploration in areas where the Transformer is confident, 2) prevent potentially failing to explore areas of high reward that the Transformer has not yet encountered.
PH-UCT is effective because its equation favors selecting an action a that 1) Q(s_t, a_t) is high, indicating the discovery of a high reward molecule. 2) \pi(a|s_t) is high, suggesting that the Transformer predicts a as a high probable next token, 3) N(s_t) is large while N(s_t, a) is small, implying that s_{t+1} has not been adequately explored. 4) the entropy of the subsequential tree is large in expectation, indicating high uncertainty in its subsequence after performing action a.
<!-- > ***Q3:*** "Alternatively, ablation studies could be done to determine which component of the ERP is important to achieve the current SOTA."
***A:*** Thank you for your suggestion. There are mainly two dimensions of variability in our new algorithm, the steps of entropy forward looking (e-steps) during selection, and the Top-P and Top-K filtering during expansion.
We note that the case of e-step=0 reduces to PG-TD or selection with P-UCB, the main algorithm our work compares with. Thus the variability of e-step subsumes the variation between selection with P-UCB and PΗ-UCB.
In Figure 3 (h) of the reviewed version of the paper, we show the effects of changing from P-UCT (i.e., e-step=0) to PΗ-UCT (i.e., e-step=1). And further improvements are seen as the e-step gets larger than 1.
Additionally, thanks to your advice, now we ablate the Top-P and Top-K filtering from our model, and summarize the new results along with a subset of the old ones in the table below. We will add the table to the future iteration of the paper.
As we can see from the table, Top-P-K filtering is important as it raises the average reward by more than 4 %. ERP with 1 step may not show significant performance across all scenarios, but raising it further to 6 is crucial in giving another 1 % improvement.
To conclude, both the Top-P-K filtering and the forward-looking multi-e-step entropy reinforcement are needed to achieve the best performance.
| E-step (Selection) | Top-P-K filtering (Expansion) | Average reward | - |
|--------------------|-------------------------------|----------------|--|
| 6 | False | 2.46 | - |
| 0 | True | 2.57 | Reduced to P-UCB, As in Figure 3 (h) |
| 1 | True | 2.57 | As in Figure 3 (h) |
| 6 | True | 2.60 | As in Figure 3 (h) |
-->
> ***Q3:*** "Table 1 could be broken into 2 rows, the current fontsize is rather small."
Thank you! We will fix it in the camera ready version.
> ***Q4:*** "how complex is the task? What is a random/negative baseline and a positive baseline? What will the scores look like?"
***A:*** Thanks to your advice, we introduced three more baselines, summarized in the table below, to help illustrate the intrinsic complexity of this particular generation problem. The results are in the table below, which will be included in the next version of our paper.
A complete-random/negative baseline is (a) sampling from a uniform distribution over the whole vocabulary. Our experiment shows that it cannot find any valid molecule over 512 trials. Thus the score is 0 in this setting.
We also consider UCT, an instantiation of MCTS, with less or completely no use of the neural LM. Thus there are two incrementally weakened algorithms. The weakest one is (b) UCT using LM in neither expansion nor evaluation, thus during expansion all tokens from the vocabulary are added and are considered equally likely. The experiment shows that no valid molecules may be found with the rollouts of 512. The other case is that (c) UCT uses LM in expansion but not in evaluation, so during the expansion stage the distribution and Top-k and Top-p filtering is used, and the more probable (according to the LM) continuation is prioritized for node selection if the scores are the same. Yet during evaluation, uniformly random sampling instead of a pre-trained LM’s prompted decoding is used. The performance of this baseline finds very few valid molecules. The table below shows experiments on 3CLPro, corresponding to the upper block of Table 1.
### Table : Extended baselines
| - | Average normalized reward | Ratio of valid molecules | Number of unique valid molecules found |
|--|---------------------------|--------------------------|----------------------------------------|
| Uniform random sampling (a) | Null | 0.0 | 0 |
| UCT without LM (b) | Null | 0.0 | 0 |
| UCT using LM in expansion only (c) | 1.99 | 0.0001 | 1 |
| Beam search with pre-trained LM (as in Table 1) | 2.38 | 1.0 | 16 |
| Sampling with pre-trained LM (as in Table 1) | 2.39 | 1.0 | 1609 |
| UCT (as in Table 1) | 2.25 | 1.0 | 3013 |
| PG-TD (as in Table 1) | 2.59 | 1.0 | 2549 |
| ERP (as in Table 1) | 2.62 | 1.0 | 2575 |
> ***Q5:*** "If an ablation study is done to show how important the PH-UCT, e-step and TOP-PK algorithms work to improve the generation, it will add to our knowledge about which component is critical."
***A:*** Thank you for your suggestion. There are mainly two dimensions of variability in our new algorithm, the steps of entropy forward looking (e-steps) during selection, and the Top-P and Top-K filtering during expansion.
We note that the case of e-step=0 reduces to PG-TD or selection with P-UCB, the main algorithm our work compares with. Thus the variability of e-step subsumes the variation between selection with P-UCB and PΗ-UCB.
In Figure 3 (h) of the reviewed version of the paper, we show the effects of changing from P-UCT (i.e., e-step=0) to PΗ-UCT (i.e., e-step=1). And further improvements are seen as the e-step gets larger than 1.
Additionally, thanks to your advice, now we ablate the Top-P and Top-K filtering from our model, and summarize the new results along with a subset of the old ones in the table below. We will add the table to the future iteration of the paper.
As we can see from the table, Top-P-K filtering is important as it raises the average reward by more than 4 %. ERP with 1 step may not show significant performance across all scenarios, but raising it further to 6 is crucial in giving another 1 % improvement.
To conclude, both the Top-P-K filtering and the forward-looking multi-e-step entropy reinforcement are needed to achieve the best performance.
### Table : Extended ablation
| E-step (Selection) | Top-P-K filtering (Expansion) | Average reward | - |
|--------------------|-------------------------------|----------------|--|
| 6 | False | 2.46 | - |
| 0 | True | 2.57 | Reduced to P-UCB, As in Figure 3 (h) |
| 1 | True | 2.57 | As in Figure 3 (h) |
| 6 | True | 2.60 | As in Figure 3 (h) |
---
We hope our responses and additional experiments have addressed your remaining concerns and have made you more confident in this work. If you have any further inquiries, please kindly let us know. Thank you!
[1] ZINC20—a free ultralarge-scale chemical database for ligand discovery, 2020
[2] DRUGIMPROVER: Utilizing Reinforcement Learning for Multi-Objective Alignment in Drug Optimization, NeurIPS Workshop on New Frontiers of AI for Drug Discovery and Development, 2023
[3] Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. "Planning with Large Language Models for Code Generation." In The Eleventh International Conference on Learning Representations. 2022.
[4] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. NeurIPS, 2021
[5] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R´emi Leblond, Tom Eccles, James Keeling, Felix Gimeno, and Agustin Dal Lago. Competition-Level Code Generation with AlphaCode. arXiv preprint arXiv:2203.07814, 2022.
# Responses to Reviewer 2fke (R1)
Thank you for the feedback on our work! Below please find our detailed responses to your questions.
---
> ***Q1:*** "In the Main Results section, how many molecules are sampled by different methods on each protein?"
***A:*** For each target/dataset, except for the beam search, the number of molecules sampled are the same across methods/rows. That is, it is the number of rollouts times and size of beams, two of the hyperparameters.
For the upper half of Table 1 in particular, we chose the hyperparameter of rolltous=256 and beam_size=16. Thus for each row, except for the beam search row, the number of molecules sampled is 256*16 = 4096.
For the lower half of Table 1, on the other hand, we chose the hyperparameter of rolltous=256 and beam_size=8. Thus for each row, except for the beam search row, the number of molecules sampled is 256*8 = 2048.
We searched for the hyperparameters based on the ranges listed in the appendix.
> ***Q2:*** Are the results reported in Table 1 the mean values? The variance should also be provided for further clarification. Figure 3 also has a similar issue. Please include the confidence intervals, as reinforcement learning is often unstable.
***A:*** Our method ERP and the baseline techniques, which include PG-TD, UCT, and Beam Search, are all deterministic. And one row in Table 1 represents only one run of the algorithm ERP, PG-TD, UCT, or beam search, rather than an average of multiple experimental runs. The same is the case in the work PG-TD [1]. Therefore, variance and confidence intervals are not applicable to our metrics. As a result, neither the results in our paper nor those in the PG-TD paper include variance or confidence intervals. As you pointed out, some of the LM's we used were fine-tuned beforehand using reinforcement learning, but we only apply our method to one such model for each target. With our deterministic search and decoding method, there are again no multiple runs to do statistical inference over. However, the Sampling baseline is stochastic, so we will include confidence intervals and variance data for this particular baseline in the next iteration of the paper. Thank you for the suggestion!
---
We hope our responses have addressed your concerns and have made you more confident in this work. If you have any further inquiries, please kindly let us know. Thank you!
[1] Planning with Large Language Models for Code Generation. ICLR 2023.
Google Drive
Gist
Clipboard
Sign in with Wallet
