Bibliography for Risk Limiting Auditing (RLA)

Bibliography for Risk Limiting Auditing (RLA) === ###### tags: `bibliography` # Papers * [A gentle introduction to risk-limiting audits](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.229.883&rep=rep1&type=pdf) * Introduces the literature for RLA * Defines key terms including *audit trail*, *risk limit*, *ballot-polling audit*, *comparison audit*, *understatement* / *undervote* and *overstatement* / *overvote* * Describes an example ballot-polling audit and an example comparison audit * Discusses methods for picking random samples of ballots, e.g. PRNGs * Provides a case study of a real-world risk limiting audit --- * [Risk-Limiting Audits](https://people.csail.mit.edu/rivest/pubs/Riv17f.pdf) * Talk given at Fourth NASEM committee meeting on Future of Voting * Briefly introduces RLA and its key terms * States possible extensions including many contests, many jurisdictions and vote-by-mail --- * [Risk-Limiting Post-Election Audits: Why and How](https://people.csail.mit.edu/rivest/pubs/RLAWG12.pdf) * Provides more detailed descriptions of the definitions and key components of a RLA * Discusses size of audits, randomness, voter privacy and choosing the risk limit * Discusses advantages and disadvantages of different RLAs --- * [Sequential tests of statistical hypothesis](https://www.jstor.org/stable/2235829) * Provides a test for null hypotheses of the form H0: p = p0 and alternative hypotheses of the form H1: p = p1 * RLAs are often based on the ratio test from this paper --- * [DiffSum - A Simple Post-Election Risk-Limiting Audit](https://people.csail.mit.edu/rivest/pubs/Riv15x.pdf) * Introduces the DiffSum ballot-polling audit * Assumes ballots are sampled without replacement * Allows an adjustable risk limit * Can be extended for multiple candidates * Not provably risk-limiting --- * [ClipAudit A Simple Post-Election Risk-Limiting Audit](https://people.csail.mit.edu/rivest/pubs/Riv17b.pdf) * Introduces the ClipAudit ballot-polling audit * Assumes ballots are sampled without replacement * Requires a beta function as a prerequisite which can be computed via simulation or approximation * Can be extended for multiple candidates and multiple winners --- * [Conservative statistical post-election audits](https://www.jstor.org/stable/30244217?seq=29#metadata_info_tab_contents) * Provides a conservative sequential test for RLA * Based on Wald's sequential ratio test * Discusses the statistics of the presented RLA method in depth --- * [Super-Simple Simultaneous Single-Ballot Risk-Limiting Audits](https://www.usenix.org/legacy/events/evt/tech/full_papers/Stark.pdf) * Introduces *simultaneous* risk limiting audits, *simultaneous risk* and *simultaneous risk limit* * Useful for elections that consist of multiple contests, e.g. election of UK Parliament and UK MPs * Requires an *error inflation factor* and *error tolerance* to control the trade-off between initial sample size and additional counting or confidence with which the error rate for the contests overall is less than the diluted margin * A Kaplan-Markov P-value is used to determine when to stop counting ballots --- * [BRAVO: Ballot-polling Risk-limiting Audits to Verify Outcomes](https://www.usenix.org/system/files/conference/evtwote12/evtwote12-final27.pdf) * Introduces the BRAVO RLA which is one of the more popular RLAs * BRAVO samples ballots with replacement and is a ballot-polling audit * Based on Wald's sequential probability ratio test * Allows for multiple candidates and multiple winners * Describes the use of a *ballot manifest* for randomly sampling ballots * Compares efficiency of pairwise comparisons between individual winners and losers against *pooling* all losers into a single loser --- * [Minerva - An Efficient Risk-Limiting Ballot Polling Audit](https://www.usenix.org/system/files/sec21-zagorski.pdf) * Introduces the Minerva RLA * Introduces *ballot-by-ballot* and *round-by-round* audits * Proven to be risk-limiting and at least as efficient as BRAVO * Minerva is modelled as a binary hypothesis test --- * [Risk-limiting Audits for Nonplurality Elections](https://apps.dtic.mil/sti/pdfs/ADA547054.pdf) * Discusses methods for auditing non-plurality elections * Modifies Stark's proposal to work for approval voting, range voting and Borda count * Provides a method for auditing both one-method and two-method Condorcet methods * Only considers Condorcet methods in a general sense, but mentions auditing for more specific methods (like the Schulze method) is an open problem * Discusses three different methods for auditing IRV elections as well as how to compute lower and upper bounds on the IRV margin * Auditing the by how much each erroneous ballot contributes to the overall margin (this is also an open problem) * Auditing the elimination order * Auditing by error detection * Looks at real-world IRV elections and their margins * [Ballot-polling Risk-limiting Audit for Selection of Condorcet Winner](https://ieeexplore.ieee.org/abstract/document/9263223) * Describes how to modify BRAVO to audit a Condorcet election * Only considers the more general Condorcet method and not the more specific ones * Requires rejection of multiple null hypotheses in the proposed method * Proposed method has an associated Python implementation * Considers average sample number for auditing Condorcet methods * Paper is missing a formal proof that the proposed method is risk-limiting * Proposed method does not consider weak Condorcet winners or indifference between rankings on a ballot --- * [VAULT: Verifiable audits using limited transparency](https://www.stat.berkeley.edu/~stark/Preprints/vault19.pdf) * Gives understandable method for a verifiable RLA * Uses homomorphic commitments and set-membership ZKP: each ballot's commitment is opened when audited * Method is an abstract framework that can be applied to any election with well-defined possible ballot values, and any RLA algorithm * The supplied RLA algorithm is inherently trusted to be correct and verifiable ---