We thank you for carefully reading our paper and providing constructive feedback. Below we provide a point-by-point response to your questions and concerns.

– The authors.

# Reviewer 4TV6

* *Response to comments on real world applications of our work.*
* API Pack aims to advance Code-LLMs for generating API calls, benefiting real-world applications such as code assistants like Github Copilot, Amazon CodeWhisperer, and LLM-powered IDE extensions.
* *Response to comments on potential biases on dataset construction.*
* *APIs selection.*
* APIs were selected from the only large-scale public OAS file sources available at the time. We'll add more details on API data, sources, and filtering to the paper and appendix.
* *Natural Language Prompts.*
* Prompts were created based on observed error patterns and existing work ([Xu et al., 2023](https://arxiv.org/abs/2304.12244)). We'll discuss prompt creation rationales and potential biases in the revised paper.
* *Programming language selection.*
* Programming languages are limited by the openapi-snippet library used. Most are in StarCoder's report ([Table 1](https://arxiv.org/abs/2402.19173)). We'll explain this and plan to expand language support in future work.
* *Answers to questions*
* *About potential risks and unintended consequences developers should be aware of utilizing LLMs trained on API Pack.*
* Thanks for highlighting this important point. We will include the *Ethical Considerations* below in the revised paper:
* Hallucination: Code generated with LLMs fine-tuned with API Pack or any other code dataset must be verified by developers.
* OAS files are from public, official API hubs. Data will be removed if legitimate owners request it. Very few OAS files contained license information. We are committed to ensure comprehensive licensing details for the next dataset version.
* No sensitive information risk: API Pack uses placeholders for argument values and types, not real values. In real applications, users provide these values through conversational AI interfaces.
* *Are there other software engineering tasks where API Pack could be beneficial?*
* API Pack focuses on API call code generation but can enhance code generation when combined with other datasets (Section 5.5).
* *Will API Pack be publicly available to the open source research community?*
* Yes! API Pack dataset, fine-tuned models, and associated codebase will be publicly available.

We have done our best to answer all your questions, we would appreciate it if you reconsider your final score.

# Reviewer snNL
* *Answers to questions*
* *Prompts used with GPT-3.5/4 are not shared:*
* We do have the prompt template in the paper (see Appendix Section, Listing 5). We will update Section 4.2 Experimental Setting to properly reference the prompts used with GPT-3.5 and 4. These prompts are the same that we used to fine-tune models. We will also add to the appendix an example prompt with real data bootstrapped into the template for better illustration.
* *Can built-in function calling capabilities of GPT-3.5/4 also generate the API call?*
* The function calling of GPT-3.5 and 4 requires pre-defined functions before execution. This differs from our scenario where users don't know which function/API endpoint to use and seek guidance from the language model to find it.
* Does training on the 1M dataset make a difference on performance?
* Yes, training on the 1M dataset improves both 0-shot and 3-shot performance in our scaling experiments, as shown in the figures below. We will update Figure 3 accordingly.
* *No performance comparison of fine-tuned Mistral and CodeLLaMA on other similar API datasets to verify that API Pack is of better quality.*
* We acknowledge this is an important experiment to address. To verify API Pack's quality, we evaluated a fine-tuned CodeLlama-13b model on 20k instances of [ToolBench dataset](https://arxiv.org/abs/2307.16789), which only contains APIs from RapidAPI. Results (see below) show that using APIs from a single source like in TollBench decreases model performance in both 0-shot and 3-shot settings compared to API Pack, which curates data from four different sources.

        |  Train    | Test   | level 1 Endpoint | level 1 API Call | level 2 Endpoint | level 2 API Call | level 3 Endpoint | level 3 API Call |
        |-----------|--------|------------------|------------------|------------------|------------------|------------------|------------------|
        | ToolBench | 0-shot | 0.057            | 0.057            | 0.081            | 0.080            | 0.073            | 0.070            |
        |           | 3-shot | 0.445            | 0.378            | 0.407            | 0.364            | 0.437            | 0.381            |
        | API Pack  | 0-shot | 0.144            | 0.103            | 0.159            | 0.133            | 0.142            | 0.089            |
        |           | 3-shot | 0.635            | 0.555            | 0.568            | 0.514            | 0.561            | 0.491            |

* Furthermore, to demonstrate the effectiveness of our data filtering pipeline, we replaced 45% of the filtered instructions with non-filtered ones and fine-tuned CodeLlama-13b with 20k instances of each. The results confirm that the 0-shot and 3-shot performances indeed drop with the introduction of non-filtered instructions.

        | Train         | Test   | level 1 Endpoint | level 1 API Call | level 2 Endpoint | level 2 API Call | level 3 Endpoint | level 3 API Call |
        |---------------|--------|------------------|------------------|------------------|------------------|------------------|------------------|
        | none-filtered | 0-shot | 0.103            | 0.083            | 0.128            | 0.103            | 0.122            | 0.084            |
        |               | 3-shot | 0.624            | 0.549            | 0.549            | 0.487            | 0.557            | 0.495            |
        | with-filtered | 0-shot | 0.144            | 0.103            | 0.159            | 0.133            | 0.142            | 0.089            |
        |               | 3-shot | 0.635            | 0.555            | 0.568            | 0.514            | 0.561            | 0.491            |

As we have addressed your concerns with additional experiments that will be integrated into the final paper, we kindly ask you to reconsider your final score.

# Reviewer aAmu
* *Answers to questions*
* *Some of the closed source information is difficult to follow i.e., "the company's API Hub", "three people (all authors of this paper)" without a verification of fluency in English or programming languages.*
* We apologize for the confusion caused by the anonymized information (e.g., company names, data hub names, URLs). We anonymized this information to adhere to the double-blind review process. We will update these details in the final version of the paper to be completely transparent with the data sources that are part of API Pack.
* *I would appreciate clarification on the multilingual terminology as programming languages. The standard meaning of "Cross-lingual transfer" to an NLP audience is different to what is used here. Frankly, I am likely only matched to review this paper owing to the keyword "multilingual" being overloaded in meanings within different ICML communities.*
* We appreciate you raising this important point about the potential confusion surrounding the term "multilingual" in our paper. We acknowledge that our usage of "multilingual" in reference to programming languages differs from the standard meaning of "cross-lingual transfer" in the NLP community. To avoid any misunderstanding, we will replace instances of "multilingual" with "multi-programming-language" and use "cross-programming-language transfer" instead of "cross-lingual transfer" in the revised version of the paper. This will help clarify the terminology and prevent confusion for readers from different ICML communities. Thank you again for the feedback.
* *Fig3: What happens to models <20k examples? Is this a minimum to observe non-negligible improvement?*
* We have expanded our analysis to include results for models trained on 10k examples, as well as the 1M total data to address this concern. The updated version of Figure 3 (see below) now showcases the performance of models trained on datasets ranging from 10k to 1M examples. The results for the 10k model align with the overall trend observed as shown in the figures in [Google Doc](https://docs.google.com/document/d/e/2PACX-1vQfvx_yQrNSgZeZMgGuEYrDScBstUVMyKf1z48wg60fS2qYJyVw1Zj4fyJQcCPmjCDW8I4nWnqRT3a9/pub).

* *Upon acceptance, would the references to "a company" and "the authors" be made clearer?*
* Yes, we will clear all anonymized information in the final version of the paper. It was written this way so as to adhere to the double-blind review process.

# Reviewer 2rkY
* *Multilinguality:*
* The multilinguality study was performed and can be found in Appendix E. To improve clarity of paper, in the revision, we will:
* Move the detailed multilinguality study from Appendix E, Figure 6 to Section 5.4.
* Expand the discussion on the importance of multi-programming-linguality.
* Emphasize how API Pack's strong performance across diverse languages sets it apart and demonstrates its practical value for multilingual applications.
* *Data curation:*
* To address this question, we conducted an experiment to compare the model's performance on the filtered and unfiltered API Pack datasets (we finetuned CodeLlama-13b using 20k instances in both cases). The results (in [Google Doc](https://docs.google.com/document/d/e/2PACX-1vQfvx_yQrNSgZeZMgGuEYrDScBstUVMyKf1z48wg60fS2qYJyVw1Zj4fyJQcCPmjCDW8I4nWnqRT3a9/pub)) confirm that the 0-shot and 3-shot performances indeed drop without the filters, validating the accuracy of our data curation process. In future work, we plan to explore additional validation methods, such as using mock API endpoints to test execution with a subset of the dataset to further ensure data quality.

* *Evaluation metrics:*
* In the definition of levels, we carefully selected and deduplicated the APIs used in training and testing. For level 1 tests, no overlapping instructions are seen during training; for level 2 tests, no overlapping API endpoints are seen; and for level 3 tests, no overlapping APIs are seen.
* The threshold of 0.9 is defined as a heuristic where the user can complete the API call based on the model response above this threshold, and we evaluate both the endpoint and API call accuracies. We acknowledge that a more granular evaluation at the per-field level could provide a more precise assessment of the model's performance. We will consider incorporating this in future work.
* *Real world alignment:*
* We acknowledge that users may not always provide the API name when querying an LLM fine-tuned with API Pack. However, in practical applications, LLMs often work with other components such as conversational UIs, retrieval mechanisms, or other LLMs. In such scenarios, models trained on API Pack can be very useful. For instance, if a user asks for an API call without providing the API name, an external mechanism may match the task to potential APIs and provide the user with a list of relevant APIs to choose from.
* *Line 027-030 right: there has been a line of work on evaluating cross-lingual performance in code tasks, like MultiPLE and MBXP. The authors should discuss these as part of related work.*
* Thanks for the suggestions. We will reference those datasets and benchmarks in the revised paper.
* *Line 126-128 left: I am not sure why this is a concern.*
* We will support this claim with a citation to [ToolLLM work](https://arxiv.org/abs/2307.16789), which explains the concern further. We will also rephrase the sentence to clarify this is a concern to the open-source research community.
* *Line 120-122 right: could the authors study how much memorization current LLMs have on these sources? Do any of the sources have license restrictions?*
* To address the question of memorization in current LLMs, we conducted additional evaluations for GPT-3.5 and GPT-4 on the level-3 0-shot setting. Our findings show that the accuracies for both models are below 1%, suggesting limited memorization of the specific API call data used in our study. Results can be found in [Google Doc](https://docs.google.com/document/d/e/2PACX-1vQfvx_yQrNSgZeZMgGuEYrDScBstUVMyKf1z48wg60fS2qYJyVw1Zj4fyJQcCPmjCDW8I4nWnqRT3a9/pub). We will address this important point in the revised paper.

* *Regarding license restrictions:* The OAS files used to build API Pack come from public and official API data hubs. API data owners published these files on those websites themselves. That said, if a legitimate API code owner requires their data to be removed from API Pack we will do so. Regarding APIs license, very few OAS files contained this information. It can be found in the API DBs as `api_license`. This attribute can be used to filter data per license if the info is available. In a future version of API Pack, we will track down license information manually for all APIs where it is missing.

* *Line 132 right: why filtering out non-English data?*
* We collected a small amount of non-English data (less than 1%) through the APIs. As it was insignificant, we removed it from our first release. We acknowledge this rationale is omitted in the paper and that we should fix it. Moreover, creating a fully multilingual expansion of API Pack is part of our project plan. We will mention it in future work too.
* *Line 189 right: what is "the company's API Hub"?*
* This is anonymized for the double-blind review, we will make the list of company names public in the final paper revision.
* *Line 293 left: "retrieved & re-ranked" maybe I missed it, but I didn't see any discussion on this part compared to 3-shot retrieved.*
* In Appendix D Table 7, we have "retrieved & re-ranked" as one of the evaluations. We will add the proper reference and discussion to the main part of the revised paper.
* *Line 291 right: what does "excluding insignificant elements" mean?*
* We apologize for the confusion. "insignificant elements" refer to specific parts of the API requests that do not affect the overall functionality or purpose of the request. Examples include:
* Whitespace and formatting differences: We ignore variations in spaces, tabs, or newlines within the API requests, as they do not change the intended behavior of the request.
* Specific naming conventions or variable names: We focus on the structure and parameters of the API requests rather than the specific names used for variables or functions, as long as they serve the same purpose.
* We will clarify this in the revised version of the paper.
* *Line 311-313 right: did you control the steps or the number of examples seen in model training? If the former, it makes sense that 3-shot finetuning will perform better, right?*
* In our experiments, we controlled the number of training examples used for fine-tuning the models.
* *Table 3: It is unclear to me what random and nearest match baselines are for these tasks?*
* The random and nearest match baselines refer to the different prompt settings used during the test time to evaluate the influence of retrieval augmentation on model generalization.
* 3-shot random: For a given test API, three API examples are randomly selected from the same API and provided to the model along with the test instruction.
* 3-shot retrieved: For each test instruction, three relevant API examples are retrieved from the same API based on their similarity to the test instruction. The three examples with the highest cosine similarity scores are selected and provided as context to the model, along with the test instruction, to generate the API call.
* In the revised paper, we will make sure to clearly define these baselines in Section 4.2 B to provide a better understanding of the experimental setup and the different prompt settings used for evaluation.

Given the additional information and the steps we have taken to strengthen our work, we kindly request that you reconsider your assessment of our research contribution.

## Reviewer xWkU

* *The one weakness that nagged me was that while this felt like a good software engineering feat, it didn't necessarily clear the bar for new research that pushes the boundaries. The volume is indeed high from the automation and selection of sources, but I don't get a real sense of diversity and value from the work overall.*
* API Pack advances the state of the art in API call code generation for the open source community by addressing the limitations of previous work, which focused on API detection intent, limited programming languages, or smaller datasets (see Table 1). Our evaluation ([Google Doc](https://docs.google.com/document/d/e/2PACX-1vQfvx_yQrNSgZeZMgGuEYrDScBstUVMyKf1z48wg60fS2qYJyVw1Zj4fyJQcCPmjCDW8I4nWnqRT3a9/pub)) demonstrates that even closed-source models like GPT-3.5 and GPT-4 perform poorly in the zero-shot setting for this task. Furthermore, we will make the API Pack dataset, fine-tuned models, and associated codebase publicly available to the research community. These contributions make our work a significant advancement in the field of API-related research.

* *How did you pick the 20k samples? What was the significance of that choice and the impact of alternate combinations?*
* In Figure 3, we studied the impact of scaling the instruction dataset size on the performance of CodeLlama-13b using zero-shot and three-shot retrieval evaluation.
* For the experiments presented in Table 3, we chose a dataset size of 20k samples mainly to save computational resources. While larger datasets lead to better performance, we found that 20k samples provided a good balance between model performance and computational efficiency.
* *How would you deal with deprecations? Does your approach already handle it given that it's likely that you might already have known deprecated APIs?*
* In our 3-shot settings, our approach is already dealing with API deprecations to some extent. The level-3 tests in our evaluation assess the model's performance on unseen APIs, demonstrating its ability to generalize to changes through Retrieval Augmented Generalization (RAG). This suggests that our approach can handle deprecations by retrieving relevant examples and generating API requests based on the available information.
* To further improve our handling of API deprecations in future iterations, we plan to regularly update the API dataset to reflect the latest changes, including deprecations, by periodically crawling API documentation sources and incorporating updates and warnings.
* *Are there annotations around performance, safety etc?*
* We do not completely understand the context of this question. However, in terms of risks and unintended consequences developers should be aware of utilizing LLMs trained on API Pack we can say:
* *Hallucination is a general risk associated with the use of LLMs.* Code generated with LLMs fine-tuned with API Pack or any other code dataset must be verified by a developer. On that note, we highlight that API Pack's code comes directly from API documentation websites, API call examples were not generated via an LLM. High-quality data has a positive impact in reducing hallucination.
* *The OAS files used to build API Pack come from public and official API data hubs.* API data owners published these files on those websites themselves. That said, if a legitimate API code owner requires their data to be removed from API Pack we will do so. Regarding API's license, very few OAS files contained this information. We acknowledge the importance of having complete and accurate license information for all APIs included in API Pack. In the future version of the dataset, we commit to tracking license information for APIs where it is currently missing.
* *There is no risk of sensitive information being released as API Pack does not include real argument values.* Instead, placeholders indicate the need for an argument value and its respective type. In a real application, the users can provide these values through a conversational AI interface (i.e., code assistant).
* *The value in using an API over writing own code (at least for pieces that can be dealt with locally) is to benefit from prior work. But if the prior work is inefficient, you might get worse results. Am I completely off base or is this deemed out of scope?*
* You raise a valid point about the potential drawbacks of using inefficient APIs. However, our work focuses on improving the accessibility and usability of APIs through LLM-generated code, rather than evaluating the efficiency of the APIs themselves. The primary value lies in saving developers time and effort by automating the process of searching for and understanding API call examples, directly impacting their productivity. While API efficiency is an important consideration, it is outside the scope of our current work, which aims to make API usage more convenient, leaving the choice and evaluation of APIs to the developers.

Given the additional information provided, we kindly request that you reconsider your assessment, as we believe our responses demonstrate the significance and novelty of our approach.