# Ollama Assurance Testing


> [!INFO] About this Note
> These note includes information about some early experiments involving Ollama and several open-source LLMs.

## Table of Contents

1. [Key Information](#key-information)
- [Models](#models)
- [Environment](#environment)
- [Assurance Case](#assurance-case)
2. [Tests](#tests)
- [Test 1](#test-1)
- [Test 2](#test-2)
## Key Information

### Models

The models used are as follows:

- Llama2 7b (`78e26419b446`): https://ollama.com/library/llama2:7b
- Mistral 7b v0.2 (`61e88e884507`): https://ollama.com/library/mistral:7b
- Gemma 7b, v1.1 (`a72c7f4d0a15`) :https://ollama.com/library/gemma:v1.1

All of these models have been downloaded from and run with [Ollama.ai](https://ollama.com/).

### Environment

All models were run using [Ollama's Docker image](https://hub.docker.com/layers/ollama/ollama/latest/images/sha256-72bffd1f1bf5744daa0668162a6cfd149efb0624db8f1886edde65c2f743b405?context=explore), with NVIDIA GPU support.

RAG was used to ensure the LLMs had access to an assurance case within its context window, and the assurance case was provided as JSON (see below).

> [!NOTE] Future Improvement
> An improvement that could be made here to enhance reproducibility would be to develop custom modelfiles for each model that include a custom prompt related to argument-based assurance and also specify key variables such as temperature to ensure fixed values.

### Assurance Case

The following assurance case was used, and provided to the model's context window as a separate document through RAG:

{
  
  "type":"AssuranceCase",
  "name":"Synthetic Data Quality (Pattern)",
  "description":"An argument pattern for assuring quality of synthetic data",
  "created_date":"2024-04-05T10:42:11.335559Z",
  "lock_uuid":null,
  "goals":[
    {
      
      "type":"TopLevelNormativeGoal",
      "name":"G1",
      "short_description":"The synthetic data generated meets the required quality standards for its intended use.",
      "long_description":"N/A",
      "keywords":"N/A",
      "assurance_case_id":66,
      "context":[],
      "property_claims":[],
      "strategies":[
        {
          
          "name":"S1",
          "short_description":"Fidelity—ensure synthetic data accurately reflects the statistical properties and distributions of the original dataset.",
          "long_description":"N/A",
          "goal_id":90,
          "property_claims":[
            {
              
              "type":"PropertyClaim",
              "name":"P1",
              "short_description":"The mean, variance, and correlation of synthetic data closely match those of the original dataset.",
              "long_description":"N/A",
              "goal_id":null,
              "property_claim_id":null,
              "level":1,
              "claim_type":"Project claim",
              "property_claims":[],
              "evidence":[],
              "strategy_id":44
            },
            {
              
              "type":"PropertyClaim",
              "name":"P2",
              "short_description":"Visual inspection of synthetic and original data distributions shows high similarity.",
              "long_description":"N/A",
              "goal_id":null,
              "property_claim_id":null,
              "level":1,
              "claim_type":"Project claim",
              "property_claims":[],
              "evidence":[],
              "strategy_id":44
            }
          ]
        },
        {
          
          "name":"S2",
          "short_description":"Diversity—ensure the synthetic data encompasses a wide range of scenarios, including rare or edge cases, to accurately represent the variability in the real dataset.",
          "long_description":"N/A",
          "goal_id":90,
          "property_claims":[
            {
              
              "type":"PropertyClaim",
              "name":"P3",
              "short_description":"The synthetic data includes representations of all known edge cases in the original dataset.",
              "long_description":"N/A",
              "goal_id":null,
              "property_claim_id":null,
              "level":1,
              "claim_type":"Project claim",
              "property_claims":[],
              "evidence":[],
              "strategy_id":45
            }
          ]
        },
        {
          
          "name":"S3",
          "short_description":"Privacy—ensure the synthetic data protects sensitive information and prevents re-identification of individuals.",
          "long_description":"N/A",
          "goal_id":90,
          "property_claims":[
            {
              
              "type":"PropertyClaim",
              "name":"P4",
              "short_description":"Synthetic data generation incorporates differential privacy techniques to minimise re-identification risks.",
              "long_description":"N/A",
              "goal_id":null,
              "property_claim_id":null,
              "level":1,
              "claim_type":"Project claim",
              "property_claims":[],
              "evidence":[],
              "strategy_id":46
            },
            {
              
              "type":"PropertyClaim",
              "name":"P5",
              "short_description":"Re-identification risk assessments confirm that the risk of identifying individuals from the synthetic data is below acceptable thresholds.",
              "long_description":"N/A",
              "goal_id":null,
              "property_claim_id":null,
              "level":1,
              "claim_type":"Project claim",
              "property_claims":[],
              "evidence":[],
              "strategy_id":46
            }
          ]
        },
        {
          
          "name":"S4",
          "short_description":"Utility—ensure the synthetic data is useful for its intended application, such as training machine learning models or analysis.",
          "long_description":"N/A",
          "goal_id":90,
          "property_claims":[
            {
              
              "type":"PropertyClaim",
              "name":"P6",
              "short_description":"Model trained on synthetic data achieves performance metrics within an acceptable range of those trained on original data.",
              "long_description":"N/A",
              "goal_id":null,
              "property_claim_id":null,
              "level":1,
              "claim_type":"Project claim",
              "property_claims":[],
              "evidence":[],
              "strategy_id":47
            },
            {
              
              "type":"PropertyClaim",
              "name":"P7",
              "short_description":"Domain experts confirm the synthetic data is realistic and meets the needs of the intended application.",
              "long_description":"N/A",
              "goal_id":null,
              "property_claim_id":null,
              "level":1,
              "claim_type":"Project claim",
              "property_claims":[],
              "evidence":[],
              "strategy_id":47
            }
          ]
        }
      ]
    }
  ],
  "owner":14,
  "edit_groups":[],
  "view_groups":[],
  "color_profile":"default",
  "comments":[],
  "permissions":"manage"
}

## Tests

### Test 1

The following prompts were asked for each model:

1. Please tell me about this assurance case.
2. Would you suggest any improvements or additions?

#### Mistral Responses

*Please tell me about this assurance case.*

> An assurance case named "Synthetic Data Quality (Pattern)" is described in the context. This assurance case is designed to ensure the quality of synthetic data generated for various uses. The case includes three sub-goals: fidelity, utility, and diversity. Fidelity focuses on ensuring that the synthetic data accurately reflects the statistical properties and distributions of the original dataset. Utility ensures that the synthetic data is useful for its intended application. Diversity ensures that the synthetic data encompasses a wide range of scenarios to represent the variability in the real dataset. The case includes several strategies and property claims under each sub-goal to achieve these objectives. For instance, under fidelity, it includes a claim stating that the mean, variance, and correlation of synthetic data closely match those of the original dataset. Similarly, under utility, it includes a claim stating that a model trained on synthetic data achieves acceptable performance metrics compared to the model trained on original data. Additionally, the case includes a visual inspection claim stating that synthetic and original data distributions show high similarity.

*Would you suggest any improvements or additions?*

> Based on the provided context, the "Synthetic Data Quality (Pattern)" assurance case appears to cover several important aspects for ensuring the quality of synthetic data, including fidelity, utility, and privacy. However, there is room for improvement or addition in terms of:
>
> 1. Expanding the scope of the sub-goals: While fidelity, utility, and privacy are essential, other factors such as robustness, completeness, consistency, and interoperability should also be considered.
> 2. Providing more concrete evidence: The context currently includes some property claims but lacks specific evidence supporting these claims. Gathering and documenting empirical evidence or simulation results would strengthen the case.
> 3. Including additional strategies for each sub-goal: There may be various ways to achieve the desired goals, so it's essential to consider multiple strategies and their trade-offs. For example, under fidelity, one strategy could be using Generative Adversarial Networks (GANs) to generate synthetic data.
> 4. Considering other relevant stakeholders: While domain experts are mentioned in the context, other stakeholders such as data scientists, regulatory bodies, and ethicists should also be involved in ensuring the quality of synthetic data.
> 5. Addressing potential risks and challenges: Synthetic data generation comes with its own set of risks and challenges, such as data leakage, biases, and ethical concerns. The assurance case should include strategies for addressing these issues.

#### Gemma Responses

*Please tell me about this assurance case.*

> The provided text contains an assurance case called "Synthetic Data Quality (Pattern)". It argues that the quality of synthetic data can be assured by verifying two key aspects: fidelity and utility. Fidelity ensures that the synthetic data accurately reflects the statistical properties of the original dataset, while utility assesses whether the data is suitable for its intended application.

*Would you suggest any improvements or additions?*

> The provided assurance case seems comprehensive in its approach to assessing the quality of synthetic data. However, there are some areas where improvements or additions could be considered:
>
> **Improvements:**
>
> - **Specificity in Fidelity:** The property claim regarding mean, variance, and correlation is a good starting point, but it could be strengthened by mentioning specific statistical measures or tests to be employed for verification.
> - **Quantitative Assessment:** Including quantitative metrics for assessing fidelity and utility would strengthen the argument. For example, including metrics like mean absolute error or reconstruction accuracy for fidelity, and metrics related to the application's performance when trained on synthetic data.
> - **Evidence Collection:** The evidence field for the property claims is currently empty. Providing relevant evidence, such as statistical reports, experimental results, or domain-specific assessments, would bolster the overall argument.
>
> **Additions:**
>
> - **Coverage of Additional Quality Dimensions:** The assurance case focuses on fidelity and utility, but other quality dimensions could be considered, such as faithfulness to domain knowledge, completeness of the synthetic data, or its ability to handle unseen scenarios.
> - **Context-Specificity:** The case could be enhanced by tailoring the fidelity and utility measures to the specific application or domain where the synthetic data is being generated.
> - **Risk Assessment:** Including a section on risk assessment, addressing potential threats to the quality of the synthetic data, would provide a more comprehensive view of the assurance case.
>
> By incorporating these suggestions, the assurance case can provide a more robust and well-rounded argument for the quality of synthetic data.

#### Llama2 Responses

*Please tell me about this assurance case.*

> Sure! Based on the provided context, here is the information you requested:
>
> The assurance case is named "Synthetic Data Quality (Pattern)" and it is an argument pattern for assuring the quality of synthetic data. The case was created on April 5, 2024, and it has 66 goals, 1 owner, and 2 edit groups.
>
> The goals of the assurance case are:
>
> - Goal 1: The synthetic data generated meets the required quality standards for its intended use.
> - Goal 90: The synthetic data incorporates differential privacy techniques to minimize re-identification risks.
>
> The property claims of the assurance case are:
>
> - Property Claim 1: The mean, variance, and correlation of synthetic data closely match those of the original dataset.
> - Property Claim 2: Visual inspection of synthetic and original data distributions shows high similarity.
> - Property Claim 3: The synthetic data includes representations of all known edge cases in the original dataset.
>
> The strategies of the assurance case are:
>
> - Strategy 44: Fidelity—ensure synthetic data accurately reflects the statistical properties and distributions of the original dataset.
> - Strategy 45: Diversity—ensure the synthetic data encompasses a wide range of scenarios, including rare or edge cases, to accurately represent the variability in the real dataset.
> - Strategy 46: Utility—ensure the synthetic data is useful for its intended application, such as training machine learning models or analysis.
>
> The evidence of the assurance case is:
>
> - Evidence 1: Domain experts confirm the synthetic data is realistic and meets the needs of the intended application.
> - Evidence 2: Visual inspection of synthetic and original data distributions shows high similarity.
>
> I hope this information helps! Let me know if you have any further questions.

*Would you suggest any improvements or additions?*

> Thank you for providing the context! Based on the information provided, I don't have any suggestions for improvements or additions to the assurance case. The case seems to be well-structured and covers all the necessary aspects of synthetic data quality. However, I would like to clarify a few things:
>
> - What is the intended application of the synthetic data? Knowing this can help tailor the property claims and strategies to better meet the needs of the application.
> - Are there any specific domains or industries that the synthetic data will be applied in? This can help inform the selection of evidence and the level of detail required for each claim.
> - Are there any other stakeholders or teams involved in the project who may have different perspectives on the quality of the synthetic data? It may be helpful to involve them in the creation and evaluation of the assurance case.
>
> Overall, I think the assurance case is a good starting point for ensuring the quality of synthetic data. With a better understanding of the project's requirements and stakeholders, we can further refine the case to meet your needs.

### Test 2

#### Mistral Responses

_Please summarise this assurance case. Focus on the top-level goal and the key strategies. Identify any relevant property claims to help explain the respective strategies._

> The assurance case discussed here is named "Synthetic Data Quality (Pattern)" and its top-level goal is that "The synthetic data generated meets the required quality standards for its intended use." This goal is achieved through three key strategies: S1, S2, and S3.
>
> Strategy S1 focuses on fidelity, ensuring that synthetic data accurately reflects the statistical properties and distributions of the original dataset. Property claims associated with this strategy include P1 stating that the mean, variance, and correlation of synthetic data closely match those of the original dataset, and P2 indicating high similarity between the synthetic and original data distributions upon visual inspection.
>
> Strategy S2 aims to ensure diversity in the synthetic data by encompassing a wide range of scenarios, including rare or edge cases, to accurately represent the variability in the real dataset. Property claim P3 supports this strategy by confirming that the synthetic data includes representations of all known edge cases from the original dataset.
>
> Strategy S3 addresses privacy concerns and ensures that the synthetic data protects sensitive information and prevents re-identification of individuals. It includes property claims P4 stating that differential privacy techniques are used to minimize re-identification risks, and P5 confirming that re-identification risk assessments show a low risk of identifying individuals from the synthetic data.

_Are there any missing property claims that could be added to Strategy S3?_

> Based on the provided context, there don't seem to be any explicitly stated missing property claims for Strategy S3 (Privacy) in the assurance case. However, it is always a good idea to consider additional property claims that could strengthen or provide more detail about the strategy. Here are a few suggestions:
>
> P8: Synthetic data does not contain any sensitive attributes or identifiable information from the original dataset (e.g., names, addresses, social security numbers).
>
> P9: Re-identification risk assessments are performed regularly to ensure that the synthetic data remains non-identifiable.
>
> P10: Synthetic data is generated using methods designed to prevent membership inference attacks (i.e., attacking the privacy of individuals by determining whether they belong to a certain group based on the synthetic data).
>
> P11: The synthetic data generation process is compliant with relevant privacy regulations, such as HIPAA, GDPR, or CCPA, depending on the domain.

_Please suggest relevant forms of evidence for all of the property claims associated with S3, including the new ones you have proposed._
_For each evidence, please list the linked property claim and say why this type of evidence is relevant._

> Based on the context provided and the additional property claims suggested, here are some potential forms of evidence for each property claim related to S3 (Privacy):
>
> 1. P4:
> - Differential privacy techniques implementation documentation or code review reports confirming usage.
> - Reports from privacy experts or auditors stating that the implemented techniques meet industry standards and best practices.
>
> 2. P5:
> - Outputs of re-identification risk assessments, such as risk scores and reports, showing that the risks are below acceptable thresholds.
> - Records of regular re-identification risk assessments being performed by a designated team or individual.
>
> 3. New property claim P8:
> - Reports from privacy experts or auditors confirming that no sensitive information is leaked in the synthetic data, based on an analysis of the dataset and comparison to original data.
>
> 4. New property claim P9:
> - Records of compliance with relevant privacy regulations, such as HIPAA, GDPR, or CCPA, and any related certifications or attestations.
>
> 5. New property claim P10:
> - Reports from users or stakeholders indicating that their privacy is respected in the synthetic data, through surveys, focus groups, or other feedback mechanisms.
>
> These types of evidence are relevant because they provide objective evidence that the synthetic data meets certain privacy standards and guidelines. Differential privacy techniques implementation documents and code reviews ensure that appropriate methods have been applied to protect individual's privacy. Re-identification risk assessments demonstrate that the risks of re-identifying individuals in the synthetic data are minimized. Compliance with relevant regulations shows that privacy best practices are being followed, while user feedback validates that those practices result in a respect for individual privacy.

#### Gemma Responses

#### Llama2 Responses