# Checklist to evaluate machine learning systems
:::info
This checklist is based on the paper [The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency and Usability in AI](https://arxiv.org/abs/2403.13784) - [DOI](https://doi.org/10.48550/arXiv.2403.13784) published Mar 21, 2024. The Model Openness Framework (MOF) is implemented on the [Model Openness Tool website](https://mot.isitopen.ai) (MOT).
:::

# Scope of this document
This Checklist was developed by volunteers during the co-design process to help reviewers of AI systems to identify and rank the components required to exercise the basic freedoms of Open Source AI. It's been further refined via public comments, on the forum and on the public draft on hackmd.

This document should be seen as part of the definitional process, a learning tool: **The Checklist is not an operating manual to evaluate Open Source AI**.

# Relationship to the Model Openness Framework

The MOF classifies systems in three degrees of availability of components, from some (Class III, Open Model) to all (Class I, Open Science). When using the MOF, one can think of the requirements of the "preferred form to make modifications to a ML system" as a bar overlayed on the MOF range of classes.

# Known issues and limitations

* **Tied to generative AI**: Being based on the MOF, this Checklist appears to be tightly coupled to generative AI. The list of components is not generalized enough to be applied to all machine learning. More research is necessary to apply the principles of the Open Source AI Definition to other kinds of AI and different machine learning systems.
* **Subject to interpretation**: When the Datasets component is made available, the Data requirements should be satisfied. When AI systems don't make the Datasets component available, one needs to extrapolate from the alternative Data components if they provide the requirements listed in the Open Source AI Definition. This is another area that requires further research as the practice of Open Source AI develops.

For more details, see also the [Open Source AI FAQ](https://opensource.org/ai/faq).

# Table of default required components

| Required components | Legal frameworks[^1] |
| ------------------------| ------------------------------ |
| **Data** |
| *See Known Issues. The requirements in the [Open Source AI Definition](https://opensource.org/ai/definition) must be satisfied.* |
| - Datasets | Available under OSI-approved terms |
| - Research paper | Available under OSI-approved terms |
| - Technical report | Available under OSI-approved terms |
| - Data card | Available under OSI-approved terms |
| **Code** |
| *All of these components are required* |
| - Data pre-processing | Available under OSI-approved license |
| - Training, validation and testing | Available under OSI-approved license |
| - Inference | Available under OSI-approved license |
| - Supporting libraries and tools | Available under OSI-approved license |
| **Model** |
| *All of these components are required* |
| - Model architecture | Available under OSI-approved license |
| - Model parameters | Available under OSI-approved terms |

# Table of optional components
The other components listed in the Model Openness Framework are optional.

| Optional components |
| ------------------------|
| **Data** |
| - Evaluation data |
| - Evaluation results |
| **Code** |
| - Code used to perform inference for benchmark tests |
| - Evaluation code |
| **Model** |
| - Model card |
| - Sample model outputs|
| - Model metadata |

[^1]: `Available under OSI-approved terms` means that the OSI will review licenses and agreements to ensure that all materials are available under terms that conform with the Open Source Definition.