---
# System prepended metadata

title: 'Autoregressive Search Engines: Generating Substrings as Document Identifiers'
tags: [筆記, NLP]

---

# Autoregressive Search Engines: Generating Substrings as Document Identifiers

###### tags: ```筆記```, ```NLP```
> https://arxiv.org/pdf/2204.10628.pdf

## Abstract
- Motivation: Addressing knowledge-intensive language tasks requires NLP systems to not only provide correct answers but also retrieve supporting evidence from a corpus. The paper proposes using autoregressive language models for generating distinctive ngrams as identifiers for **passage-level retrieval**, offering a new approach to improve search accuracy **without imposing a predefined structure** on the search space.
    - 這邊指不用DSI方法中去assign semantically structured identifiers
- Results: The proposed method, leveraging all ngrams in a passage as identifiers, outperforms prior autoregressive approaches and established retrieval solutions on the KILT benchmark by at least 10 points, setting new state-of-the-art performances on some datasets with a significantly lighter memory footprint.

## Introduction
- Knowledge-intensive tasks in NLP, such as open-domain question answering and fact-checking, often combine a search engine with a machine reader to retrieve relevant information and generate answers. The surge in autoregressive language models has boosted machine reader performance but not equally improved retrieval methods.
- The paper introduces an autoregressive model that uses all ngrams in a passage as possible identifiers, eliminating the need for a structured search space and leveraging an efficient data structure to map generated ngrams to full passages.

## Methodology
- Method Name: Search Engines with Autoregressive LMs (SEAL).
    - ![image](https://hackmd.io/_uploads/r1TzFaFRT.png)
- SEAL combines an autoregressive model (BART) with a compressed full-text substring index (FM-Index) for generating and scoring distinctive ngrams that are mapped to passages. This approach allows for the generation of any span from the corpus **without explicitly encoding all substrings**, using a novel scoring function that combines LM probabilities with FM-index frequencies to improve retrieval accuracy.

## Experiments
- Datasets: Evaluation conducted on **Natural Questions** (NQ) and the **[KILT benchmark](https://paperswithcode.com/dataset/kilt)**, involving various knowledge-intensive tasks across multiple datasets.
- Metrics: Accuracy@k for NQ, R-precision for KILT benchmark at passage level, and comparison of memory footprint sizes. SEAL demonstrates improved performance over established retrieval solutions and previous autoregressive approaches, achieving new state-of-the-art downstream results on multiple datasets when combined with existing reader technologies.


> The contents shared herein are quoted verbatim from the original author and are intended solely for personal note-taking and reference purposes following a thorough reading. Any interpretation or annotation provided is strictly personal and does not claim to reflect the author's intended meaning or context.