# 2.1. Evaluation Framework

[](https://hackmd.io/ZOR0K6IsQXieqTZNJASAQA)

Technological solutions to medical problems always walk the fine line between progress and safety. The history of medicine is unfortunately littered with well-intended but deadly innovations. Accelerated design, manufacturing and clinical deployment of safe, novel, complex medical devices during a global public health emergency, with the attendant stresses on our infrastructure, presents additional and distinct challenges to those that routinely confront developers of healthcare technologies. Most prominent among these are first, the securing of essential human and material resources for development and manufacturing, and second, the assurance of a level of safety and usability that would ensure the device does more good than harm to patients and the highly stressed workers and institutions that care for them. In the face of global pandemic like COVID-19 a new movement has emerged to address the shortage of ventilators. Presenting as a particularly striking example given the urgency, technical and physiologic complexity, medical uncertainty, the large range of potential hazards for patients and operators and the vast amounts of financial resources made available to address the problem.

Many ventilator initiatives during the COVID 19 pandemic have sought to address the resource challenge by adopting open-source intellectual property models in order to leverage mass collaboration and crowdsourcing in the development and manufacturing process. This has been very effective in rapidly mobilizing a large number of contributors with access to diverse expertise and resources and has led to a vast array of projects exploring the problem space.

The second challenge, that of ensuring a basic level of safety and usability, has received far less attention in the public domain, but has been the subject of intense efforts by agencies tasked with patient safety. Confronted with the challenge of adapting a complex and multi-faceted evaluation process that typically takes 2-5 years to complete and which varies significantly by jurisdiction, agencies around the globe have provided various guidance documents and standards for emergency use ventilators in order to support accelerated temporary licensing while seeking to minimize potential hazards to the public. Health Canada (HC), UK’s Medicines and Healthcare products Regulatory Agency (MHRA), U.S. Food and Drug Administration (FDA), Australia’s Therapeutic Goods Administration (TGA) have all released documents to this effect. This has been a major step forward. There remains however a significant gap in the application of these jurisdiction specific, variously detailed and sometimes divergent documents, to trans-national open source projects that have significant medical and engineering expertise but often lack a deep understanding of patient safety standards and the regulatory processes that implement them.

We seek to bridge this gap by synthesizing the available guidance provided by the above-mentioned organizations into a unified framework for the evaluation of emergency use ventilators. The technical evaluation component of the framework is applicable to all emergency use ventilators regardless of the intellectual property licensing of the project. However, given the number of nominally open source projects in the space we have also included a section to pragmatically evaluate the openness of projects based on best practices recommended by the Open Source Hardware Association (OSHWA). This document is intended for use by developers working on specific projects, health care system administrators and staff evaluating candidate devices, and working to prepare providers for their eventual deployment and clinical use.

Ventilators currently in clinical use comprise a range of devices optimized for various use cases, time courses and settings such as treatment of acute respiratory distress syndrome in the intensive care unit, ventilation of surgical patients in the operating room, short term ventilation of patients during transport or off-site diagnostic testing and long-term ventilation of patients with chronic neuromuscular diseases. This paper focuses specifically on emergency use ventilators. [Formal definition is provided by any of the agencies]. These devices are intended as temporary, last-resort devices for situations where standard, approved devices are not accessible. They are required to provide minimum viable functionality and safety features and be optimized for simplicity and rapid manufacturing.

The emergence of open source ventilator designs over the course of COVID-19 pandemic development resulted in a number of attempts to derive tools that can help classify and rate these designs based on their openness, readiness, and deployment viability. One of the most recent works that showcases this trend was published by Joshua M. Pearce, [“A review of open source ventilators for COVID -19 and future pandemics”](https://f1000research.com/articles/9-218). In this article, Mr. Pearce is discussing designs and development proposed by a number of teams that work on open source ventilation systems. He points out work supervised by [Robert Read et al., whose team is currently keeping track of all globally developed emergency ventilation projects](https://docs.google.com/spreadsheets/d/1inYw5H4RiL0AC_J9vPWzJxXCdlkMLPBRdPgEVKF8DZw/edit#gid=0) while providing a high-level ranking of each project in terms of openness, buildability, community support, functional testing, reliability, COVID-19 suitability, and clinical friendliness. The initiative aims to provide a score on a scale of 1-5 for each respective category of assessment. This initiative, which provided a framework for tracking and high-level evaluation of designs that have achieved a reasonable level of development, can be found on GitHub along with the color-coded spreadsheet that contains all accompanying information.

Providing such high-level overview, many open source ventilator projects databases do not discuss the regulatory assessment as part of the ranking or validation process. While this general assessment of open source files and performance metrics can be used as a basis for evaluation of validity and suitability for further research and development, it fundamentally does not allow the user to define whether the product has the potential to achieve regulatory approval status by the organizations such as Health Canada, MHRA, FDA, TGA, and others.

To address this gap, our team has focused on the development of an expansive evaluation framework that is applicable to design and development of emergency ventilation systems at the time of global pandemic and encompasses three levels of evaluation that allow the used to navigate complex pathway towards regulatory approval
___

## C. Open Source Design Evaluation Framework: Levels of Evaluation

The flow of the proposed assessment is determined by the precedence of data necessity. To evaluate a design, we must first check the available information, then assess its performance, and determine the compliance to all other requirements.

### Level 1 – Open Source Documentation:
> Level one contains open access assessment in accordance with what is required for the build of the proposed design project. Assessment includes overview of licensing, general information, and accompanying documentation that must be shared by the designer in order to allow external replication and future development. The assessment checklist used in this section was derived based on the availability of guidelines around the assessment of openness based on OSHWA standards. Our group found that despite a comprehensive evaluation checklist, OSHWA recommendations do not account for the evaluation of open source documents with the regulatory compliance in mind. Thus, any documentation presented to showcase testing data must be accompanied with comprehensive explanations and support. We found that without availability of testing data, Level 2 and 3 evaluation can be significantly inhibited and incomplete. We believe that in order to describe best practices around sharing and maintenance of open source design information, more comprehensive research must be conducted around this topic.

### Level 2 - Performance Parameters:

>Level two provides guidance for the assessment of general performance capabilities and parameter check which must be attained by any ventilator to provide adequate patient support. These parameters shall be satisfied irrespective of the mechanism or design used. This section was derived based on the suggestions outlined by Health Canada, FDA, MHAR, and TGA interim orders. The proposed checklist includes a list of minimum viable performance requirements and is not intended to be used for evaluation of complexity of the assessed device. We recognize that different systems are built with specific intention in mind. However, in order to provide minimum support in an emergency situation, ventilator must be able to satisfy minimum requirements as outlined by the abovementioned regulatory agencies.

### Level 3 - Compliance with International Standards:

> Level three is a combination of checklists and standards that the user must adhere to obtain device approval by Health Canada, FDA, MHRA, and TGA. This assessment level provides guidelines necessary for the evaluation of design’s safety and compliance with interim orders and regulatory standards specific to the pandemic period. This level is further broken down into sections that evaluate the designs from the perspective of hardware, software, mechanical and electrical systems, risk, labeling and human factors. Each criterion for the evaluation includes a comprehensive checklist that allows the user to follow regulatory standard requirements and associated tests.