### Meeting Notes

#### NeurIPS push
- fine-tuning for SNLI,

paper todo:
- write like red/green watermark paper
- with DPO: only need logits from modeller!!!!!
- experiments that show how easy it is to cheat!: use this for inspiration https://arxiv.org/pdf/2306.04634.pdf

- try to refute impossibility result here: https://arxiv.org/pdf/2304.11082.pdf
- run an experiment here showing that we can mitigate sleeper agents: https://arxiv.org/pdf/2401.05566.pdf
- response: what about sleeper agent in adaptor?
- adaptor is from trusted party (regulator)

13/02/24
- spin:
- we have a lot of methods, not enough ideas of how to do alignment in practice other than: "companies you should do this!"
- https://arxiv.org/pdf/2304.11082.pdf

02/01/24
- red/green list training method: https://arxiv.org/pdf/2305.15065.pdf
- to make more interpretable: https://arxiv.org/pdf/2204.10628.pdf

26/01/24
- red/green model
- desiderata:
- way more light-weight than LLM
- sentence by sentence
- training ideas:
- parameter efficient fine tuning: https://github.com/huggingface/peft
- paper: Autoregressive Search Engines: https://arxiv.org/abs/2204.10628
- training data:
- https://github.com/anthropics/hh-rlhf

12/01/24
- watermark should be such that if you mess with a valid model then it becomes invalid
- is there an algorithm to test whether after a change, a watermark has been removed?
- baseline:
- certification is benchmarking
- idea: run all benchmarks (assume company can't identify benchmarks)
- when model is open, problem becomes trivial, just test with a bunch of different people
- when regulator has model people can check output with them, but requires a bunch of compute
- IDEA:
- person supplies TEXT, number
- whenever the number is special e.g., sum of TEXT numbers, then the output of the model is also sum of that number
- ?: How do we ensure model has this behavior!?
- could have a separate script
- ?: can we make this script undecipherable by the regulator?
- NO!
- what if user and modeller always engage in MPC so that modeller can never see final output?
- Modeller supplies initial output, then user does MPC with that output to verify checks hold
- still need to make sure model has this behavior!!!
papers

### New idea:

**Sketch:** Regulator has set of inputs $\mathbf{X} = \{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$ that produces outputs $\mathbf{Y} = \{\mathbf{y}_1, \ldots, \mathbf{y}_n\}$, and can prove model has changed just by seeing if outputs change.

#### Assumptions
- Modeler cannot tell if the model is being used by a User or Regulator
- Modeler does not recognize any of $\{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$ as special
<!-- - Only non-linearity in model is RELU$(\cdot) := \max\{0, \cdot\} := [\cdot]_+$ (likely easy incorporate other things like batch normalization, pooling, dropout)

#### Network
-->
We have an $L$ layer network with weights $\mathbf{W}^{(i)} \in \mathbb{R}^{w}$ and biases $\mathbf{b}^{(i)} \in \mathbb{R}^b$. To map input $\hat{\mathbf{x}}^{(0)} \in \mathbb{R}^{\textrm{in}}$ to output $\mathbf{x}^{(L)} \in \mathbb{R}^{\textrm{out}}$ we apply the following functions alternately:

$$ \mathbf{x}^{(i)} = \mathbf{W}^{(i)}\hat{\mathbf{x}}^{(i-1)} + \mathbf{b}^{(i)} \\
\hat{\mathbf{x}}^{(i)} = \max(0, \mathbf{x}^{(i)})$$

(we can think of softmax as part of the loss function not the network)
(likely easy incorporate other things like batch normalization, pooling, dropout)

#### Problem statement

Given $\mathbf{X},\mathbf{Y}$ and a constraint on HERERERE: https://arxiv.org/pdf/2302.01404.pdf

$$ \max_\delta \|\delta\| \;\;\; \mbox{s.t.,} \; [\theta+\delta](\mathbf{x}_i) = \mathbf{o}_i, \; \forall i \in \{1, \ldots, n\}$$

### Goal

Predictions that are guaranteed to come from a certified model:

#### assumptions
- Regulator can modify $\theta$ to any $\theta'$ so long as certain metrics aren't harmed (much).

#### things we want to avoid
- Full verified computation of $\theta(x)$, as $\theta$ has billions of parameters and the functions we need to evaluate are non-linear: 

### Possible Solutions

#### Sensitive Samples

*Idea:* Regulator has set of inputs $\{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$ that produces outputs $\{\mathbf{o}_1, \ldots, \mathbf{o}_n\}$, and can check if model has changed by seeing if outputs change.

*Additional Assumptions:*
- Inputs $\{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$ and outputs $\{\mathbf{o}_1, \ldots, \mathbf{o}_n\}$ are only known to Regulator
- Modeler cannot tell if the model is being used by a User or Regulator

*Guarantee Sketch:*
- The set of unnoticed perturbations $P$ becomes smaller as the size of the secret set grows.

*Current Issues:*
- Every input $\mathbf{x}$ produces a non-zero activation from every weight, however it is technically possible change the model $\theta$ so that the same output $\mathbf{o}$ is sampled (i.e., if the final probability distribution is largely unchanged).
- Reduces to solving a high-dimensional optimization problem: $$ \max_\delta \|\delta\| \;\;\; \mbox{s.t.,} \; [\theta+\delta](\mathbf{x}_i) = \mathbf{o}_i, \; \forall i \in \{1, \ldots, n\}$$
If the optimum is 0 then we are done. But solving this problem for LLMs is hugely expensive. Can we approximate it? Note this is super related to certified adversarial robustness:
- https://arxiv.org/pdf/2206.10550.pdf
- https://arxiv.org/pdf/2104.06718.pdf
- Need to find an efficient upper bound!

#### Watermarking

*Idea:* Regulator can change $\theta$ to $\theta'$ so that (a) it can identify this change from the model outputs and (b) the Modeler cannot. As above we will use a special set of inputs (not outputs).

*Additional Assumptions:*
- Inputs $\{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$ are only known to Regulator

*Guarantee Sketch:*
- Regulator has a proof regulator has swapped in a different model, proof does not reveal details of how regulator was able to check this.

*Current Issues:*
- Say Regulator does watermarking during training in the following way (inspired by https://arxiv.org/pdf/2301.10226.pdf).
- It takes a trained model $\theta$ from the Modeller and fine-tunes it to $\theta'$
- This works by first selecting a set of random words $w_1, \ldots, w_n$ and random probabilities $p_1, \ldots, p_1$. It then changes the model as little as possible so that for inputs $\mathbf{x}_i$ the above random words have above random probabilities:
$$ \theta' := \theta + {\arg\!\min}_{\delta} \|\delta\| \;\;\; \mbox{s.t.,} \; \mathbb{P}\Big([\theta+\delta](\mathbf{x}_i) = w_i \Big) = p_i, \; \forall i \in \{1, \ldots, n\}$$
- This is definitely do-able, and the words can be chosen from a list so they do not harm accuracy much. Also this seems hard to detect by the Modeler (they could technically search for the inputs $\{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$ by looking for where the probabilities have changed and then modify the model while keeping the changed probabilities fixed. But! If we add similar random noise to every other probability then we are good). The main open question is how to make a guarantee out of this? Could we get a DP-style guarantee?

### Other Ideas / Papers
- backdoor insertion:
- https://arxiv.org/pdf/2204.06974.pdf
- only works on simple models currently
- homomorphic max
- precomputed points (checkpoints) on which you know the output of the model
- bind the checkpoints with the new point on which you have to evaluate any new point
- idea: can you break up the computation so you can get proofs for the intermediate steps
- regulator-specific language
- LLMs speak paralell language from hashing the vocab with a particular key
- Output english, weird language: results from hashing at training
- check weird language using key regulator provided

### related papers

#### VeriDL: Integrity Verification of Outsourced Deep Learning Services (Extended Version)
- https://arxiv.org/pdf/2107.00495.pdf
- **method**: full verification
- can't find proofs anywhere
- limited experiments

#### Validating the integrity of Convolutional Neural Network predictions based on Zero-Knowledge Proof
- https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4216545
- **method**: full verification
- CNN specific
- not practical (experiments just with single layers)

#### DeepiSign: Invisible Fragile Watermark to Protect the Integrity and Authenticity of CNN
- https://dl.acm.org/doi/pdf/10.1145/3412841.3441970?casa_token=IMzETd7OkxwAAAAA:ntLr1Gy2V4scV6mW9AnSqzTVCxPSus4NnLhSOyQMUp6GMhJmVVDXec2-UdUG7ZOXfJTS6UHdVfAi
- **method**: watermarking
- CNN specific
- only empirical

#### pvCNN: Privacy-Preserving and Verifiable Convolutional Neural Network Testing
- https://ieeexplore.ieee.org/document/10086653
- **method**: full verification
- tools: HE, zk-SNARK
- CNN specific

#### Robust and Lossless Fingerprinting of Deep Neural Networks via Pooled Membership Inference
- https://arxiv.org/pdf/2209.04113.pdf
- **method**: training dataset inference
- just tested on image classification

#### NeuNAC: A novel fragile watermarking algorithm for integrity protection of neural networks
- https://www.sciencedirect.com/science/article/pii/S0020025521006642
- **method**: watermarking
- only empirical

#### Secure and Verifiable Inference in Deep Neural Networks
- https://dl.acm.org/doi/pdf/10.1145/3427228.3427232?casa_token=Lu0IyGyGwosAAAAA:Nlb1vBeaK8RN87bzdo4H5TIcXvsoYd6v0sgluzwU7RnRQwEdwPwPIIQKCVVLLqeMvMKcbajL-EJ4
- **method**: sensitive samples
- only empirical
- requires swapping nonlinearities with polynomials

#### A Watermark for Large Language Models
- John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, Tom Goldstein
- https://arxiv.org/pdf/2301.10226.pdf
- **method**: watermarking
- watermarks at prediction time, so not tied to a specific certified model

#### Provable Robust Watermarking for AI-Generated Text
- https://arxiv.org/pdf/2306.17439.pdf
- **method**: watermarking
- follow-up on the above paper, still not tied to a specific certified model

#### Watermarking Text Generated by Black-Box Language Models
- https://arxiv.org/pdf/2305.08883.pdf
- **method**: watermarking
- another follow-up that does not require white-box access to model to watermark, still not tied to certified model

#### Regulating ChatGPT and other Large Generative AI Models
- https://dl.acm.org/doi/pdf/10.1145/3593013.3594067
- motivation paper

#### A Recipe for Watermarking Diffusion Models
- https://arxiv.org/pdf/2303.10137.pdf
- **method**: watermarking
- trains a watermark by altering training data using an encoder and pre-training a detector
- downside: _watermark is identifiable by anyone_ so company can try to adjust model but preserve watermark

#### Robust Multi-bit Natural Language Watermarking through Invariant Features
- https://arxiv.org/pdf/2305.01904.pdf
- **method**: watermarking
- posthoc word replacement
- no guarantees