# Community Survey - Analysis Plan
:::info
This documents outline an analysis plan for the [Community Pulse Check on Assurance of digital twins](https://www.turing.ac.uk/community-pulse-check-assurance-digital-twin-systems)
To get more details on the questions please consult the additional documentation:
- [Live Survey](https://teadt.azurewebsites.net/)
- [Survey Design Document](https://thealanturininstitute.sharepoint.com/:w:/r/sites/tea-dh/_layouts/15/Doc.aspx?sourcedoc=%7B4D6879F9-38AA-4659-A4DC-95C817D034B8%7D&file=TEA-DT_community_survey.docx&action=default&mobileredirect=true&wdsle=0)
- [Sharepoint folder (internal access only)](https://thealanturininstitute.sharepoint.com/sites/tea-dh)
:::
## Questions suitable for segmenting
:::info
A couple of questions, especially those from section 1 on "Community Composition" will be used as grouping or control factors in the analysis, i.e.:
- Use `sector` as a grouping factor but take into account `location`, `role`, `primary_responsiblities` to control for sample imbalances within sector groupings
- `role` and `primary_responsbilities` can be used as groupings for the insights on Needs.
- `purpose_dt`, `type_dt` and `data_sharing` should be taken into account when looking at relevance ratings for gemini principles as the type & purpose of digital twin as well as requirement to share data will correlate with relevance of certain principles
:::
| Question Code| Type | Notes |
|-------------|------|-------|
| sector | Multiple Choice | Different sectors may have established different preferred assurance mechanisms & different levels of maturity in assurance practices
| location | Multiple Choice | *control for sampling bias* Jurisdictions may differ in vocabulary hence confounding selection of assurance mechanisms & different levels of maturity in assurance practices
| role | Multiple Choice |*control for sampling bias* Respondents with different roles may only have selected awareness of the assurance mechanisms used. Can be used to group identified needs
| primary_responsibilities | Select all that apply |*control for sampling bias* Respondents with different responsibilities may only have selected awareness of the assurance mechanisms used. Can be used to group identified needs
| established_dt | Yes/No | Having established a digital twin can indicate greater maturity and experience and potentially result in broader portfolio of assurance mechansisms.
| type_dt | Multiple Choice | No prior expectations
| purpose_dt | Multiple Choice |If answer is 'public impact' and/or 'Customer Engagement', we should expect greater adoption of assurance around ethical principles
| asset_data_sharing | Yes/No | *control for sampling bias* Those who share data with partners more likely to assure for: "Data Quality", "Data Stewardship", "Federation", "Interoperability".
## Section 1: Current Practices
:::info
RQ1: - What are the current assurance practices implemented in the digital twinning space.
:::
| Question | Question Code | Type | Scale |
|------------------|---------------|------|-------|
| Which of the following assurance mechanisms do you currently rely on for your own (or your client's) digital twin(s)? |assurance_mechanisms | Select all that apply |Nominal |
Which of the following properties (or goals) do you currently consider when assuring your (or your client’s) digital twinning technology| assured_properties | Select all that apply | Nominal | Frequency per sector
:::info
Responses can be coded into Technical, Operational or Ethical focus. Though some answers also describe broader assurance mechanisms (i.e. Bias Assessment)
Group by:
- sector
- location
:::
````mermaid
graph LR
%% Questions and Grouping
A[Group by sector]
A --> C[assurance_mechanisms]
A --> D[assurance_properties]
%% Analysis
C --> E[Frequency mechanisms]
D --> E[Frequency properties]
C --> G1[Technical Focus]
D --> G1
C --> G2[Operational Focus]
D --> G2
D --> G3[Ethical Focus]
G1 --> F[Ratio of Technical to non-technical]
G2 --> F[Ratio of Technical to non-technical]
G3 --> F[Ratio of Technical to non-technical]
%% Styling the Question Codes
style C fill:#b3cde0,stroke:#333,stroke-width:2px
style D fill:#b3cde0,stroke:#333,stroke-width:2px
%% Styling the Endpoints
style F fill:#77dd77,stroke:#333,stroke-width:2px
style E fill:#77dd77,stroke:#333,stroke-width:2px
%% Styling the Groupings
style G1 fill:#ffb3ba,stroke:#333,stroke-width:2px
style G2 fill:#ffb3ba,stroke:#333,stroke-width:2px
style G3 fill:#ffb3ba,stroke:#333,stroke-width:2px
%% Legend
subgraph Legend
L1[Question Codes]
L2[Response Coding]
L3[Reports]
end
style L1 fill:#b3cde0,stroke:#333,stroke-width:2px
style L2 fill:#ffb3ba,stroke:#333,stroke-width:2px
style L3 fill:#77dd77,stroke:#333,stroke-width:2px
````
:::warning
:face_with_monocle: **Need Feedback**
- Do we need to combine sector groups as we have a very high level of granularity atm (e.g. Technology and Innovation,Industrial and Manufacturing,Energy and Environment,Government and Public Services,Infrastructure and Logistics )
- Even so, we will not have a large enough sample to achieve equal samples per sector group.
- How well can we infer a technical focus from the types of assurance mechanisms reported?
:::
## Section 2a: Maturity/Effectiveness of Assurance Practices
:::info
RQ2: - How effectively are assurance practices integrated in the broader project lifecycle?
:::
| Question | Question Code | Type | Scale |
|------------------|---------------|------|-------|
|How do you currently communicate your assurance strategies to stakeholders, partner organisations or your clients?| communication_methods | Select all that apply | Nominal |
|Our assurance activities extend beyond mere checklist compliance and are substantively integrated into our (or our client's) operational practices.| integrate_assurance | Likert | Ordinal
|To what extent did you rely/are you relying on communication of assurance mechanisms (such as those selected in Q2.2) to establish trust between the parties of a connected digital twin ecosystem?**\*** |reliance_on_evidence | Likert |Nominal
**\*** *This question is conditional, we expect data from only approx. 50% of respondents.*
````mermaid
graph LR
%% Questions and Grouping
A[Survey Responses]
A --> B1[communication_methods]
A --> B2[integrate_assurance]
A --> B3[reliance_on_evidence]
%% Classification Based on Responses
B1 --> C1[Not Systematic, Non-standardized Reports]
B1 --> C2[Standardized Reports]
B1 --> C3[Interactive Platforms, Visual Aids]
C1 --> D1[Ineffective]
C2 --> D2[Intermediate]
C3 --> D3[Effective]
%% Combining Likert Responses
B2 --> E1[Likert Scale Response]
B3 --> E2[Likert Scale Response]
E1 --> F1[Low Integration]
E1 --> F2[Medium Integration]
E1 --> F3[High Integration]
E2 --> F1
E2 --> F2
E2 --> F3
%% Intermediate Classification
D1 --> I1
F1 --> I1
D2 --> I2
F2 --> I2
D3 --> I3
F3 --> I3
I1[Any scored as Low Integration] --> H1[Relatively Ineffective]
I2[Any scored as Intermediate and none as Low Integration] --> H2[Some Intermediate]
I3[None scored as No Low or Intermediate] --> H3[Highly Effective]
%% Styling the Question Codes
style B1 fill:#b3cde0,stroke:#333,stroke-width:2px
style B2 fill:#b3cde0,stroke:#333,stroke-width:2px
style B3 fill:#b3cde0,stroke:#333,stroke-width:2px
%% Styling the Classifications
style D1 fill:#ffb3ba,stroke:#333,stroke-width:2px
style D2 fill:#ffb3ba,stroke:#333,stroke-width:2px
style D3 fill:#ffb3ba,stroke:#333,stroke-width:2px
%% Styling the Intermediate and Endpoints
style I1 fill:#ffb3ba,stroke:#333,stroke-width:2px
style I2 fill:#ffb3ba,stroke:#333,stroke-width:2px
style I3 fill:#ffb3ba,stroke:#333,stroke-width:2px
style H1 fill:#77dd77,stroke:#333,stroke-width:2px
style H2 fill:#77dd77,stroke:#333,stroke-width:2px
style H3 fill:#77dd77,stroke:#333,stroke-width:2px
%% Legend
subgraph Legend
L1[Question Codes]
L2[Response Coding]
L3[Reports]
end
style L1 fill:#b3cde0,stroke:#333,stroke-width:2px
style L2 fill:#ffb3ba,stroke:#333,stroke-width:2px
style L3 fill:#77dd77,stroke:#333,stroke-width:2px
````
:::info
This section could be analyzed by sector or without sub-grouping depending on sample size/bias due to imbalance.
Summary insights can be made about overall percentage of respondents whose integration of the assurance process with broader practices is ineffective/effective, which can then be explained as being due to "e.g a lack of systematic communication methods"
:::
:::warning
:face_with_monocle: **Need Feedback**
- Unsure about how to code free text question on assurance meaning:
- Q:What do you understand ‘assurance’ to mean in the context of your work in the digital twinning sector?
- Is it fair to summarize an orgs assurance process as broadly "ineffective" based on these few specific questions?
- Alternatively could simply report on Frequency for each question.
:::
## Section 2a: Satisfaction
:::info
RQ3: How satisfied are practitioners with their current assurance practices?
:::
| Question | Question Code | Type | Scale |
|------------------|---------------|------|-------|
| The way we communicate our assurance activities significantly contributes to building and maintaining trust and confidence among our (or our client's) stakeholders.| communication_impact | Likert Scale 1-5 | Ordinal
|We can clearly link specific assurance activities directly to higher-level principles guiding our (or our client's) system's trustworthiness and ethical standards| link_assurance_activities | Likert Scale 1-5 | Ordinal
| How satisfied are you with how your team identifies and documents requirements, actions, and decisions in your assurance process?| satisfaction_justification | Likert Scale 1-5 | Ordinal
````mermaid
graph LR
%% Questions and Grouping
A[Survey Responses]
A --> B1[communication_impact]
A --> B2[link_assurance_activities]
A --> B3[satisfaction_justification]
%% Frequency Reporting
B1 --> F1[Per Question Frequency Low/Neutral/High]
B2 --> F1
B3 --> F1
%% Aggregated Score Calculation
B1 --> D[Aggregate Score]
B2 --> D
B3 --> D
%% Classification Based on Aggregate Score
D --> E1[Low Satisfaction 3-7]
D --> E2[Medium Satisfaction 8-11]
D --> E3[High Satisfaction 12-15]
%% Styling the Question Codes
style B1 fill:#b3cde0,stroke:#333,stroke-width:2px
style B2 fill:#b3cde0,stroke:#333,stroke-width:2px
style B3 fill:#b3cde0,stroke:#333,stroke-width:2px
%% Styling the Classifications
style E1 fill:#77dd77,stroke:#333,stroke-width:2px
style E2 fill:#77dd77,stroke:#333,stroke-width:2px
style E3 fill:#77dd77,stroke:#333,stroke-width:2px
%% Frequency Reporting Styling
style F1 fill:#77dd77,stroke:#333,stroke-width:2px
%% Styling the Endpoints
style D fill:#77dd77,stroke:#333,stroke-width:2px
%% Legend
subgraph Legend
L1[Question Codes]
L3[Aggregate Score and Frequency Reporting]
end
style L1 fill:#b3cde0,stroke:#333,stroke-width:2px
style L3 fill:#77dd77,stroke:#333,stroke-width:2px
````
:::warning
:face_with_monocle:
How to deal with lack of specificity of section 3 responses?
- Where people have answered 'stongly agree' or 'strongly disagree', we could follow-up with a semi-structured interview.
- Those questions that are rated as low satisfaction may also provide information about needs to some degree
:::
## Challenges & Needs: High-level Principles
:::info
RQ4: Which trustworthy or ethical principles are challenging to assure, and what are the specific challenges?
:::
| Question | Question Code | Type |
|------------------|---------------|------|
|Have you considered sharing data or models with other organisations (or across partners within an organisation)to form connected digital twins?| asset_data_sharing | Yes/No |
|When partnering with other organisations for building connected digital twins, how difficult is it to establish trust in the competence, goodwill and value of the resulting shared digital twin?**\***| partner_trust_difficulty | Likert |
|What are/were the major challenges to overcome? (when partnering for connected digital twins)**\***| partner_trust_challenges | Select all that apply |
Please rate, for each of the following Gemini principles, the extent to which it focuses on issues that you believe to be relevant for your work | relevance_of_principles | Likert |
Please rate, for each of the following Gemini principles: How challenging is it to define and/or to know how to currently address it in practices? | challenge_in_application | Likert |
|What are the main challenges you (or your clients) face? [with respect to the gemini principles]| operationalization_challenges | Free text|
:::info
- **Quadarant Analysis** identifying which principles are rated as highly relevant & challenging. This analysis would best be segmented by `sector` and `purpose_dt`
- Reporting **Frequency** of orgs for which data sharing is a requirement.
- Reporting **Frequency** of orgs that struggle to establish trust with partners
- **Topic modelling** of free text responses to characterize specific challenges voiced.
:::
## Readiness & Needs: ABA Tool
:::info
RQ5: Which sectors & stakeholders have high potential for adopting the TEA-DT tool, i.e have a need for it, see its value and have the conditions to adopt it. And what support is needed to drive adoption?
:::
| Question | Question Code | Type |
|------------------|---------------|------|
|How valuable do you find high-level guiding principles in general? | value_of_guiding_principles | Likert
|Who provides assurance within/for your organisation? | assurance_experience | multiple Choice
|Would a tool that helps you structure and communicate how your assurance measures align with key ethical goals enhance trust in your digital twin(s)?|need_for_visual_tool | Yes/No/I don't know | Nominal
|What do you believe are the main benefits?| benefits_of_visual_tool **\*** | Free Text |
|Why not?| reasons_against_visual_tool **\*** | Free Text|
|How prepared do you feel to link your current assurance activities with broader in a structured argument?|preparedness_for_argument | Likert |
|Does your organisation have an established definition or framework for 'trustworthy' and 'ethical' digital twins?| ethical_framework_existence | Yes/No/Don't know |
|What would prevent you from adopting a new trustworthy and ethical assurance tool?| challenges_adoption| Select all that apply|
|What type of support might help you in creating sound assurance arguments around ethical principles for your digital twin project? |support_for_assurance| Select all that apply
````mermaid
graph LR
%% Questions and Grouping
A[assurance_experience]
C2 --> B2[need_for_visual_tool]
C2 --> B3[benefits_of_visual_tool]
C2 --> B4[reasons_against_visual_tool]
C2 --> B5[preparedness_for_argument]
%% Frequency Calculations
A --> C1[Frequency of orgs using third party assurers]
A --> C2[Include In-house assurance, group by sector/role]
B2 --> D1[Frequency of orgs reporting yes or I don't know]
B5 --> E1[Frequency of subjective preparedness judgment]
%% Topic Modeling
B3 --> G1[Topic modeling of benefits]
B4 --> G2[Topic modeling of reasons against]
%% Styling the Question Codes
style A fill:#b3cde0,stroke:#333,stroke-width:2px
style B2 fill:#b3cde0,stroke:#333,stroke-width:2px
style B3 fill:#b3cde0,stroke:#333,stroke-width:2px
style B4 fill:#b3cde0,stroke:#333,stroke-width:2px
style B5 fill:#b3cde0,stroke:#333,stroke-width:2px
style C2 fill:#ffb3ba,stroke:#333,stroke-width:2px
style C1 fill:#77dd77,stroke:#333,stroke-width:2px
style D1 fill:#77dd77,stroke:#333,stroke-width:2px
style E1 fill:#77dd77,stroke:#333,stroke-width:2px
style G1 fill:#77dd77,stroke:#333,stroke-width:2px
style G2 fill:#77dd77,stroke:#333,stroke-width:2px
%% Legend
subgraph Legend
L1[Question Codes]
L2[Reporting]
L3[Response Coding]
end
style L1 fill:#b3cde0,stroke:#333,stroke-width:2px
style L2 fill:#77dd77,stroke:#333,stroke-width:2px
style L3 fill:#ffb3ba,stroke:#333,stroke-width:2px
````
:::info
- **Frequency** of orgs using third party assurers, then exclude those that do
- Group by `sector` and/or `role`
- **Frequency** of orgs reporting yes or I don't know
- **Frequency** of subjective preparedness judgment
- **Topic modelling** of free text responses
:::
:::warning
:face_with_monocle: **Need Feedback**
- We have invited free text descriptions of the ethical frameworks (or urls) if they have reported to have one, how can we best analyze that data? (Note: so far none of turing-external have reported)
- extract principles / overlap with gemini principles
:::
### Other Questions
| Question | Question Code | Type |
|------------------|---------------|------|
|What is the main reason? [for not having established a twin] | no_dt_reason | Select all that apply |
|How familiar are you with the Gemini principles? | familiarity_with_gemini_principles | Multiple Choice |
|Is there anything else pertinent to the assurance of digital twins, which has not been covered, that you think is significant? | additional_insights | Free text
## All Question
| Number | Question Code | Question | Response Options |
|--------|---------------|----------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1.1 | Q1.1 | In which country is your organisation registered? | Global, United Kingdom, [List of all countries] |
| 1.2 | Q1.2 | What is your role within your organisation? | Senior Management (e.g., CEO, CFO, CTO), Strategic/Business Lead (e.g., Business Unit Head, Project/Program Manager, Operations Manager), Strategic Advisor (e.g., Compliance Officer, Regulatory Affairs Manager, Legal Advisor), In-house Technical Specialist (e.g., Developer/Engineer, Data Scientist/Analyst, IT Specialist), Digital Twin Support Specialist (e.g., Ontology Developer, Dashboard Developer, Data Integration Specialist), Consultant/External Specialist (e.g., Industry Consultant, External IT Consultant, Freelance Technical Expert), Research and Development (e.g., Researcher/Academic, Innovation Specialist, Product Development Scientist), Other (Please specify) |
| 1.3 | Q1.3 | What are your primary responsibilities? | Strategic Direction, Budget Management, Ensuring Compliance, Project Leadership, Technical Decision-Making, Operational Management, Governance Influence, Research and Innovation, Other (Please specify) |
| 1.4 | Q1.4 | What sector best represents your field of work? | Aerospace, Architecture, Artificial Intelligence, Automotive, Aviation, Construction, Consumer Goods, Defence, Education, Electronics, Engineering, Environment and Conservation, Finance, Food and Agriculture, Freight, Healthcare, Information technology / Software, International Government, Local Government, Manufacturing, Maritime, Media, Mining, National Government, Energy, Oil and Gas, Place Leadership, Rail, Smart Cities, Supply Chain and Logistics, Technology, Telecommunications, Transport, Utilities, Waste and Recycling, Water, Other |
| 1.5 | Q1.5 | Has your organisation established one or more Digital Twins? | Yes, No, Indirectly (We support clients or provide components for digital twins) |
| 1.5b | Q1.5b | [If Yes or Indirectly] What type of digital twin? Please Select all that apply | System, Place/Infrastructure Digital Twin, Product, Process, Physical asset, Other (Please specify) |
| 1.5c | Q1.5c | [If Yes or Indirectly] Which of the following best describes the purpose of the digital twin(s) you have (helped) establish? | Internal Operations, External Collaboration, Public Impact, Market / Customer Engagement, Research and Development, Other |
| 1.5d | Q1.5d | [If No] What is the main reason? | Unclear business case/ROI, insufficient digital awareness/skills, lack of goodwill/competence/data assurance trust in partner organisation, concerns rea confidentiality of data, IPR, regulatory concerns re DP, info security, legal agreement barriers, Other |
### Section 2: Current Assurance Practices and Understanding
| Number | Question Code | Question | Response Options |
|--------|---------------|--------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 2.1 | Q2.1 | What do you understand ‘assurance’ to mean in the context of your work? | Open-ended response |
| 2.2 | Q2.2 | Which of the following assurance mechanisms do you currently rely on? | Bias Reviews, Compliance Audits, Conformity Assessments, Stakeholder Feedback Systems, Impact Assessment, Risk Assessment, Information Security Reviews, Data Quality Checks, Formal Verification, Post-Implementation Evaluation, Service Continuity Management, Performance Monitoring, Operational Audits, Service Quality Reviews, Other (Please specify) |
| 2.3 | Q2.3 | [If Q2.2 not empty] Which of the following properties do you currently consider when assuring your digital twinning technology? | Accountability, Contestability, Data Quality, Data Stewardship, Ethical Integrity, Evolution, Explainability, Fairness, Federation, Financial Performance, Fit-for-purpose, Governance, Interoperability, Openness, Public Good, Reliability/Robustness, Resilience / Fault-tolerance, Safety, Security, Sustainability, Transparency, Trustworthiness, Value Creation, None, Other (Please Specify) |
| 2.4 | Q2.4 | Who provides assurance within/for your organisation? | In-house assurance team, In-house, non-specialized team, External third-party, We provide assurance services for clients |
| 2.5 | Q2.5 | Have you considered sharing your asset-related data or models with other organisations to form connected digital twins? | Yes, No |
| 2.5b | Q2.5b | [If Yes] When partnering with other organisations for building connected digital twins, how difficult is it to establish trust? | Very Easy, Somewhat Easy, Neutral, Somewhat Difficult, Very Difficult (+ not applicable) |
| 2.5c | Q2.5c | [If 2.5b selected as more difficult or neutral than easy] What are/were the major challenges to overcome? | Multiple Purpose |
### Section 3: Satisfaction with Assurance Practices
| Number | Question Code | Question | Response Options |
|--------|---------------|--------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 3.1 | Q3.1 | To what extent do you agree with the following statements: | Strongly disagree, Disagree, Neutral, Agree, Strongly agree |
| 3.2 | Q3.2 | How satisfied are you currently with justification and documentation around your assurance process? | Very unsatisfied, Somewhat unsatisfied, Neutral, Somewhat satisfied, Very satisfied |
### Section 4: High-Level Assurance Goals and Ethical Frameworks
| Number | Question Code | Question | Response Options |
|--------|---------------|------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 4.1 | Q4.1 | Does your organisation have an established definition or framework for "trustworthy" and "ethical" digital twins? | Yes, No, I don’t know |
| 4.2 | Q4.2 | [If yes or something similar] If your framework is publicly available, please provide a link URL | Open-ended response |
| 4.3 | Q4.3 | How was this definition or framework developed? | Consensus-based process, Internal governance process, Developed by an external consultant, Reused/adapted existing framework/standards |
| 4.4 | Q4.4 | How valuable do you find high-level guiding principles in general? | Not valuable at all, Slightly valuable, Moderately valuable, Very valuable, Extremely valuable |
| 4.6 | Q4.6 | How familiar are you with the Gemini principles? | Unfamiliar, Slightly Familiar, Somewhat Familiar, Familiar, Expert |
| 4.7 | Q4.7 | Please rate, for each Gemini principle individually, the extent to which it focuses on issues that you believe to be relevant for your work | Not Relevant, Slightly Relevant, Moderately Relevant, Very Relevant, Extremely Relevant |
| 4.8 | Q4.8 | For each of the following ethical principles, rate how challenging you find it to determine if you have adequately addressed the principle in practice | Not challenging at all, Slightly challenging, Moderately challenging, Very challenging, Extremely challenging |
| 4.9 | Q4.9 | [Open-ended] For those rated very difficult or extremely difficult to operationalize, what are the main challenges you face? | Open-ended response |
### Section 5: Communicating Assurance
| Number | Question Code | Question | Response Options |
|--------|---------------|----------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 5.1 | Q5.1 | How do you currently communicate your project’s assurance strategies to your stakeholders or partner organisations? | Written Reports following established standards, Non-standardized written Reports, Meetings, Visual Aids, Digital Communications, Interactive Platforms, Not Systematically / ad hoc, Other |
| 5.2 | Q5.2 | Would a visual tool that helps you demonstrate and communicate how your evidence-based assurance measures align with key ethical goals enhance trust in your digital twin(s)? | Yes, No, I need to know more about this tool to decide |
| 5.2b | Q5.2b | [If yes] What do you believe are the main benefits? | Open-ended response |
| 5.2c | Q5.2c | [If no] Why not? | Open-ended response |
| 5.3 | Q5.3 | How prepared do you feel to develop a structured argument for how your current assurance activities relate to broader ethical goals? | Very prepared, Somewhat prepared, Neutral, Somewhat unprepared, Not prepared at all |
| 5.4 | Q5.4 | What would prevent you from adopting a new trustworthy and ethical assurance tool? | Doesn’t fit into our governance process, Doesn’t integrate into our tech stack, Internal resistance, No time to spend on ethical assurance, Cost prohibitive, Lack of
## Content analysis
| Number | Question Code | Question | Research Question for Content Analysis|
|--------|---------------|---------------|------------|
| 2.1 | assurance_meaning | What do you understand ‘assurance’ to mean in the context of your work? | Is there a shared understanding of assurance in the community? How mature is this understanding of assurance?
| 4.9 | operationalization_challenges | For those rated very difficult or extremely difficult to operationalize, what are the main challenges you face? | What is the challenge & why is it challenging? Can this challenge be addressed by TEA?
| 5.2b |benefits_of_visual_tool|Would a visual tool that helps you demonstrate and communicate how your evidence-based assurance measures align with key ethical goals enhance trust in your digital twin(s)? **If yes** What do you believe are the main benefits?| What is the unique selling point / appeal of TEA? What problems is the TEA tool solving for people?
| 5.2c |reasons_against_visual_tool | Would a visual tool that helps you demonstrate and communicate how your evidence-based assurance measures align with key ethical goals enhance trust in your digital twin(s)?**If no**, Why not?| Lack of data, ignore for now.
- Strategy:
- stage 1: Nathan & Sophie to create initial codes
- stage 2: all apply codes
- stage 3: all extract topics
- Method:
- working in excel add new column for notes
- Break up responses where necessary
- Highlight keywords, differentiate between respondent wording & coder interpretation
- Apply codes
### Codes: Assurance meaning
- **assurance as process**: For responses describing assurance as a process or series of actions or tasks.
- **assurance as outcome**: responses describing assurance as what it achieves, rather than what it is (the DT is xy, rather than testing if it isxy)
- **example of process**: for responses highlighting specific components of the rpocess (eg structured)
- **user trust/confidence**: For responses emphasizing building or maintaining user trust.
- **pragmatic adequacy**: For responses emphasizing degree of outcome in context of use
- **validation/testing/verification**: For responses mentioning a process of testing and using test results
- **compliance/standards**: For responses emphasizing adherence to standards or regulations.
- **risk awareness**: For responses highlighting the identification and mitigation of risks or unwanted harms or safety issues
- **intended outcomes of DT**: For responses verifying that system features align with intended requirements and claims.
- **lifecycle**: For responses considering the entire lifecycle of the system or process.
- **objectivity**: Responses that highlight the importance of impartiality and unbiased evaluation.
- **stakeholder involvement**: any mention of who assurance is for and how invovled they are.
- **validity to real world**: Specific about digital twins & their relationship to physical systems
- **unsure**: "I don't know" type of responses
- **data concerns** : eg data quality
- **reliability**
- **accuracy**
- **privacy/security**
- **utility/value**
- **explainability/understandable**
- **interoperability**
- **responsibility**
- **transparency**
- **accessiblity**
### Codes for Benefits for tool
**Increased trust**: trust as an beneficial outcome from applying the tool
**Improved understanding**: application of tool leads to improved understanding internally and/or externally of both system itself and assurance process.
**Standardised approach**: application of the tool leading to more standardised approaches to assurance
**Stakeholder engagement/communication**: responses mentioning use of/involvement with stakeholders
**Caveats**: response lists caveats in addition to / instead of benefits
**Structured argument**: response specifically alludes to benefits of a structured argument
**Challenging/revisiting assurance**: discussion of ability to question the argument
**Usability**: positive affirmation of usability of the tool
### Codes for Reasons against tool
**Doesn't apply**: respondent does not deem tool suitable for their products
**Unrealistic/feasibility**: scepticism about this tool being realised
**Communication of Tool**: tool does not meet communication needs
**Integration**: scepticism about the tool integrating with current workflows
### Codes: Operationalising challenges
- **data concerns**: eg data availability, data quantity
- **technology changes**: rapid evolution of tech
- **consistency**: concistency in application and developing DTs
- **appropriate people involved / lack of expertise**
- **balancing openness vs privacy**: client interest
- **short-term vs long-term interests**
- **explaining/communicating value**
- **lack of community consensus**: eg accepted standards, coordination, immature sector
- **contextual/sector-specific concerns**
- **lack of time/budget**
- **changing/adding to organisational processes**: eg how to add new processes in existing systems, getting buy-in
- **how to assure DTs**: eg lack of metrics/KPIs
- **lack of definitions**
### Aug 15th Meeting
- Nathan to review how to compute reliability
- Sophie to ask in Turing chat for recommendations re ICR
- Chris suggestion:
- coding each tag as how consistent it was used across coders to get ranking of tags
- going forward using: collab - notebook
method going forward:
- 1) identify qualitative differences: we chose a) different tags, b) additional codes
- 2) identify degree of difference: how many coders differ in their codes
- 3) identify if difference is between code creators, independent coder or both
- 4) extend code definitions, 'missed code', merge codes, add codes
- compute IRR
- then worked out 'more established' code-book
- then find aligned code
## Coding Note
### assurance meaning
- discussion point on how some responses are more detailed and 'definition-like' compared to others more intuitive or anecdotal
- codes are hard to find, that's because community is not aligned on their understanding on assurance
## Planned Outputs
- Slidedeck for early conclusion in July
- Report co-authored with DTHub (eg paper with working group)
- 'safety' concerns are not particularly prominent in the assurance meaning section. This is significant, as safety assurance is the foundation for a lot of ours and York's work so the fact that it's not a common code/theme is worth mentioning.
- Use TEA-DT budget to get illustrations / graphics for report
Sign in with Wallet
Connect another wallet