---
title: Machine Learning Software Standards
tags: statistical-software
robots: noindex, nofollow
---
<!-- Edit the .Rmd not the .md file -->
## Machine Learning Software
R has an extensive and diverse ecosystem of Machine Learning (ML)
software which is very well described in the corresponding [CRAN Task
View](https://cran.r-project.org/web/views/MachineLearning.html). Unlike
most other categories of statistical software considered here, the
primary distinguishing feature of ML software is not (necessarily or
directly) algorithmic, rather pertains to a *workflow* typical of
machine learning tasks. In particular, we consider ML software to
approach data analysis via the two primary steps of:
1. Passing a set of *training* data to an algorithm in order to
generate a candidate mapping between that data and some form of
pre-specified output or response variable. Such mappings will be
referred to here as “models”, with a single analysis of a single set
of training data generating one model.
2. Passing a set of test data to the model(s) generated by the first
step in order to derive some measure of predictive accuracy for that
model.
A single ML task generally yields two distinct outputs:
1. The model derived in the first of the previous steps; and
2. Associated statistics of model performance, as evaluated within the
context of the test data used to assess that performance.
Click on the following link to view a demonstration [Application of
Machine Learning Software
Standards](https://hackmd.io/Ix1YwD8YTWGuzdiXsVQadA).
**A Machine Learning Workflow**
Given those initial considerations, we now attempt the difficult task of
envisioning a typical standard workflow for inherently diverse ML
software. The following workflow ought to be considered an “extensive”
workflow, with shorter versions, and correspondingly more restricted
sets of standards, possible dependent upon envisioned areas of
application. For example, the workflow presumes input data to be too
large to be stored as a single entity in local memory. Adaptation to
situations in which all training data can be loaded into memory may mean
that some of the following workflow stages, and therefore corresponding
standards, may not apply.
Just as typical workflows are potentially very diverse, so are outputs
of ML software, which depend on areas of application and intended
purpose of software. The following refers to the “desired output” of ML
software, a phrase which is intentionally left non-specific, but which
it intended to connote any and all forms of “response variable” and
other “pre-specified outputs” such as categorical labels or validation
data, along with outputs which may not necessarily be able to be
pre-specified in simple uni- or multi-variate form, such as measures of
distance between sets of training and validation data.
Such “desired outputs” are presumed to be quantified in terms of a
“loss” or “cost” function (hereafter, simply “loss function”)
quantifying some measure of distance between a model estimate (resulting
from applying the model to one or more components of a training data
set) and a pre-defined “valid” output (during training), or a test data
set (following training).
Given the foregoing considerations, we consider a typical ML workflow to
progress through (at least some of) the following steps:
1. ***Input Data Specification*** Obtain a local copy of input data,
often as multiple *objects* (either on-disk or in memory) in some
suitably structured form such as in a series of sub-directories or
accompanied by additional data defining the structural properties of
input objects. Regardless of form, multiple objects are commonly
given generic labels which distinguish between `training` and `test`
data, along with optional additional categories and labels such as
`validation` data used, for example, to determine accuracy of models
applied to training data yet prior to testing.
2. ***Pre-Processing*** Define transformations of input data, including
but not restricted to, broadcasting dimensions (as defined below)
and standardising data ranges (typically to defined values of mean
and standard deviation).
3. ***Model and Algorithm Specification*** Specify the model and
associated processes which will be applied to map the input data on
to the desired output. This step minimally includes the following
distinct stages (generally in no particular order):
1. Specify the kind of model which will be applied to the training
data. ML software often allows the use of pre-trained models, in
which case this this step includes downloading or otherwise
obtaining a pre-trained model, along with specification of which
aspects of those models are to be modified through application
to a particular set of training and validation data.
2. Specify the kind of algorithm which will be used to explore the
search space (for example some kind of gradient descent
algorithm), along with parameters controlling how that algorithm
will be applied (for example a learning rate, as defined above).
3. Specify the kind of loss function will be used to quantify
distance between model estimates and desired output.
4. ***Model Training*** Apply the specified model to the training data
to generate a series of estimates from the specified loss function.
This stage may also include specifying parameters such as stopping
or exit criteria, and parameters controlling batch processing of
input data. Moreover, this stage may involve retaining some of the
following additional data:
1. Potential “pre-processing” stages such as initial estimates of
optimal learning rates (see above).
2. Details of summaries of actual paths taken through the search
space towards convergence on local or global minimum.
5. ***Model Output and Performance*** Measure the performance of the
trained model when applied to the test data set, generally requiring
the specification of a metric of model performance or accuracy.
Importantly, ML workflows may be partly iterative. This may in turn
potentially confound distinctions between training and test data, and
accordingly confound expectations commonly placed upon statistical
analyses of statistical independence of response variables. ML routines
such as cross-validation repeatedly (re-)partition data between training
and test sets. Resultant models can then not be considered to have been
developed through application to any single set of truly “independent”
data. In the context of the standards that follow, these considerations
admit a potential lack of clarity in any notional categorical
distinction between training and test data, and between model
specification and training.
The preceding workflow mentioned a couple of concepts the
interpretations of which in the context of these standards may be seen
by clicking on the corresponding items below. Following that, we proceed
to standards for ML software, enumerated and developed with reference to
the preceding workflow steps. In order that the following standards
initially adhere to the enumeration of workflow steps given above, more
general standards pertaining to aspects such as documentation and
testing are given following the initial five “workflow” standards.
<details>
<summary>
Click for a definition of *broadcasting*, referred to in Step 2, above.
</summary>
<p>
The following definition comes from a vignette for the [`rray`
package](https://github.com/r-lib/rray) named
[*Broadcasting*](https://rray.r-lib.org/articles/broadcasting.html).
- ***Broadcasting*** is, “repeating the dimensions of one object to
match the dimensions of another.”
This concept runs counter to aspects of standards in other categories,
which often suggest that functions should error when passed input
objects which do not have commensurate dimensions. Broadcasting is a
pre-processing step which enables objects with incommensurate dimensions
to be dimensionally reconciled.
The following demonstration is taken directly from the [`rray`
package](https://github.com/r-lib/rray) (which is not currently on
CRAN).
``` r
library (rray)
a <- array(c(1, 2), dim = c(2, 1))
b <- array(c(3, 4), dim = c(1, 2))
# rbind (a, b) # error!
rray_bind (a, b, .axis = 1)
#> [,1] [,2]
#> [1,] 1 1
#> [2,] 2 2
#> [3,] 3 4
rray_bind (a, b, .axis = 2)
#> [,1] [,2] [,3]
#> [1,] 1 3 4
#> [2,] 2 3 4
```
Broadcasting is commonly employed in ML software because it enables ML
operations to be implemented on objects with incommensurate dimensions.
One example is image analysis, in which training data may all be
dimensionally commensurate, yet test images may have different
dimensions. Broadcasting allows data to be submitted to ML routines
regardless of potentially incommensurate dimensions.
</p>
</details>
<details>
<summary>
Click for a definition of *learning rate*, referred to in Step 5, above.
</summary>
<p>
- ***Learning Rate*** (generally) determines the step size used to
search for local optima as a fraction of the local gradient.
This parameter is particularly important for training ML algorithms like
neural networks, the results of which can be very sensitive to
variations in learning rates. A useful overview of the importance of
learning rates, and a useful approach to automatically determining
appropriate values, is given in [this blog
post](https://sgugger.github.io/how-do-you-find-a-good-learning-rate.html).
</p>
</details>
<br>
Partly because of widespread and current relevance, the category of
Machine Learning software is one for which there have been other notable
attempts to develop standards. A particularly useful reference is the
[MLPerf organization](https://www.mlperf.org/) which, among other
activities, hosts several [github
repositories](https://github.com/mlperf) providing reference datasets
and benchmark conditions for comparing performance aspects of ML
software. While such reference or benchmark standards are not explicitly
referred to in the current version of the following standards, we expect
them to be gradually adapted and incorporated as we start to apply and
refine our standards in application to software submitted to our review
system.
### 1 Input Data Specification
Many of the following standards refer to the labelling of input data as
“testing” or “training” data, along with potentially additional labels
such as “validation” data. In regard to such labelling, the following
two standards apply,
- **ML1.0** *Documentation should make a clear conceptual distinction
between training and test data (even where such may ultimately be
confounded as described above.)*
- **ML1.0a** *Where these terms are ultimately eschewed, these
should nevertheless be used in initial documentation, along with
clear explanation of, and justification for, alternative
terminology.*
- **ML1.1** *Absent clear justification for alternative design
decisions, input data should be expected to be labelled “test”,
“training”, and, where applicable, “validation” data.*
- **ML1.1a** *The presence and use of these labels should be
explicitly confirmed via pre-processing steps (and tested in
accordance with **ML7.0**, below).*
- **ML1.1b** *Matches to expected labels should be
case-insensitive and based on partial matching such that, for
example, “Test”, “test”, or “testing” should all suffice.*
The following three standards (**ML1.2**–**ML1.4**) represent three
possible design intentions for ML software. Only one of these three will
generally be applicable to any one piece of software, although it is
nevertheless possible that more than one of these standards may apply.
The first of these three standards applies to ML software which is
intended to process, or capable of processing, input data as a single
(generally tabular) object.
- **ML1.2** *Training and test data sets for ML software should be
able to be input as a single, generally tabular, data object, with
the training and test data distinguished either by*
- *A specified variable containing, for example, `TRUE`/`FALSE` or
`0`/`1` values, or which uses some other system such as missing
(`NA`) values to denote test data); and/or*
- *An additional parameter designating case or row numbers, or
labels of test data.*
The second of these three standards applies to ML software which is
intended to process, or capable of processing, input data represented as
multiple objects which exist in local memory.
- **ML1.3** *Input data should be clearly partitioned between training
and test data (for example, through having each passed as a distinct
`list` item), or should enable an additional means of categorically
distinguishing training from test data (such as via an additional
parameter which provides explicit labels). Where applicable,
distinction of validation and any other data should also accord with
this standard.*
The third of these three standards for data input applies to ML software
for which data are expected to be input as references to multiple
external objects, generally expected to be read from either local or
remote connections.
- **ML1.4** *Training and test data sets, along with other necessary
components such as validation data sets, should be stored in their
own distinctly labelled sub-directories (for distinct files), or
according to an explicit and distinct labelling scheme (for example,
for database connections). Labelling should in all cases adhere to
**ML1.1**, above.*
The following standard applies to all ML software regardless of the
applicability or otherwise of the preceding three standards.
- **ML1.5** *ML software should implement a single function which
summarises the contents of test and training (and other) data sets,
minimally including counts of numbers of cases, records, or files,
and potentially extending to tables or summaries of file or data
types, sizes, and other information (such as unique hashes for each
component).*
#### 1.1 Missing Values
Missing data are handled differently by different ML routines, and it is
also difficult to suggest generally applicable standards for
pre-processing missing values in ML software. The [*General
Standards*](#general-standards) for missing values (**G2.13**–**G2.16**)
do not apply to Machine Learning software, in the place of which the
following standards attempt to cover a practical range of typical
approaches and applications.
- **ML1.6** *ML software which does not admit missing values, and
which expects no missing values, should implement explicit
pre-processing routines to identify whether data has any missing
values, and should generally error appropriately and informatively
when passed data with missing values. In addition, ML software which
does not admit missing values should:*
- **ML1.6a** *Explain why missing values are not admitted.*
- **ML1.6b** *Provide explicit examples (in function
documentation, vignettes, or both) for how missing values may be
imputed, rather than simply discarded.*
- **ML1.7** *ML software which admits missing values should clearly
document how such values are processed.*
- **ML1.7a** *Where missing values are imputed, software should
offer multiple user-defined ways to impute missing data.*
- **ML1.7b** *Where missing values are imputed, the precise
imputation steps should also be explicitly documented, either in
tests (see **ML7.2** below), function documentation, or
vignettes.*
- **ML1.8** *ML software should enable equal treatment of missing
values for both training and test data, with optional user ability
to control application to either one or both.*
### 2 Pre-processing
As reflected in the workflow envisioned at the outset, ML software
operates somewhat differently to statistical software in many other
categories. In particular, ML software often requires explicit
specification of a workflow, including specification of input data (as
per the standards of the preceding sub-section), and of both
transformations and statistical models to be applied to those data. This
section of standards refers exclusively to the transformation of input
data as a pre-processing step prior to any specification of, or
submission to, actual models.
- **ML2.0** *A dedicated function should enable pre-processing steps
to be defined and parametrized.*
- **ML2.0a** *That function should return an object which can be
directly submitted to a specified model (see section 3, below).*
- **ML2.0b** *Absent explicit justification otherwise, that return
object should have a defined class minimally intended to
implement a default `print` method which summarizes the input
data set (as per **ML1.5** above) and associated transformations
(see the following standard).*
Standards for most other categories of statistical software suggest that
pre-processing routines should ensure that input data sets are
commensurate, for example, through having equal numbers of cases or
rows. In contrast, ML software is commonly intended to accept input data
which can not be guaranteed to be dimensionally commensurate, such as
software intended to process rectangular image files which may be of
different sizes.
- **ML2.1** *ML software which uses broadcasting to reconcile
dimensionally incommensurate input data should offer an ability to
at least optionally record transformations applied to each input
file.*
Beyond broadcasting and dimensional transformations, the following
standards apply to the pre-processing stages of ML software.
- **ML2.2** *ML software which requires or relies upon numeric
transformations of input data (such as change in mean values or
variances) should allow optimal explicit specification of target
values, rather than restricting transformations to default generic
values only (such as transformations to z-scores).*
- **ML2.2a** *Where the parameters have default values, reasons
for those particular defaults should be explicitly described.*
- **ML2.2b** *Any extended documentation (such as vignettes) which
demonstrates the use of explicit values for numeric
transformations should explicitly describe why particular values
are used.*
For all transformations applied to input data, whether of dimension
(**ML2.1**) or scale (**ML2.2**),
- **ML2.3** *The values associated with all transformations should be
recorded in the object returned by the function described in the
preceding standard (**ML2.0**).*
- **ML2.4** *Default values of all transformations should be
explicitly documented, both in documentation of parameters where
appropriate (such as for numeric transformations), and in extended
documentation such as vignettes.*
- **ML2.5** *ML software should provide options to bypass or otherwise
switch off all default transformations.*
- **ML2.6** *Where transformations are implemented via distinct
functions, these should be exported to a package’s namespace so they
can be applied in other contexts.*
- **ML2.7** *Where possible, documentation should be provided for how
transformations may be reversed. For example, documentation may
demonstrate how the values retained via **ML2.3**, above, can be
used along with transformations either exported via **ML2.6** or
otherwise exemplified in demonstration code to independently
transform data, and then to reverse those transformations.*
### 3 Model and Algorithm Specification
A “model” in the context of ML software is understood to be a means of
specifying a mapping between input and output data, generally applied to
training and validation data. Model specification is the step of
specifying *how* such a mapping is to be constructed. The specification
of *what* the values of such a model actually are occurs through
training the model, and is described in the following sub-section. These
standards also refer to *control parameters* which specify how models
are trained. These parameters commonly include values specifying numbers
of iterations, training rates, and parameters controlling algorithmic
processes such as re-sampling or cross-validation.
- **ML3.0** *Model specification should be implemented as a distinct
stage subsequent to specification of pre-processing routines (see
Section 2, above) and prior to actual model fitting or training (see
Section 4, below). In particular,*
- **ML3.0a** *A dedicated function should enable models to be
specified without actually fitting or training them, or if this
(**ML3**) and the following (**ML4**) stages are controlled by a
single function, that function should have a parameter enabling
models to be specified yet not fitted (for example,
`nofit = FALSE`).*
- **ML3.0b** *That function should accept as input the objects
produced by the previous Input Data Specification stage, and
defined according to **ML2.0**, above.*
- **ML3.0c** *The function described above (**ML3.0a**) should
return an object which can be directly trained as described in
the following sub-section (**ML4**).*
- **ML3.0d** *That return object should have a defined class
minimally intended to implement a default `print` method which
summarises the model specification, including values of all
relevant parameters.*
- **ML3.1** *ML software should allow the use of both untrained
models, specified through model parameters only, as well as
pre-trained models. Use of the latter commonly entails an ability to
submit a previously-trained model object to the function defined
according to **ML3.0a**, above.*
- **ML3.2** *ML software should enable different models to be applied
to the object specifying data inputs and transformations (see
sub-sections 1–2, above) without needing to re-define those
preceding steps.*
A function fulfilling **ML3.0–3.2** might, for example, permit the
following arguments:
1. `data`: Input data specification constructed according to **ML1**
2. `model`: An optional previously-trained model
3. `control`: A list of parameters controlling how the model algorithm
is to be applied during the subsequent training phase (**ML4**).
A function with the arguments defined above would fulfil the preceding
three standards, because the `data` stage would represent the output of
**ML1**, while the `model` stage would allow for different pre-trained
models to be submitted using the same data and associated specifications
(**ML3.1**). The provision of a separate `.data` argument would fulfil
**ML3.2** by allowing one or both `model` or `control` parameters to be
re-defined while submitting the same `data` object.
- **ML3.3** *Where ML software implements its own distinct classes of
model objects, the properties and behaviours of those specific
classes of objects should be explicitly compared with objects
produced by other ML software. In particular, where possible, ML
software should provide extended documentation (as vignettes or
equivalent) comparing model objects with those from other ML
software, noting both unique abilities and restrictions of any
implemented classes.*
- **ML3.4** *Where training rates are used, ML software should provide
explicit documentation both in all functions which use training
rates, and in extended form such as vignettes, of the importance of,
and/or sensitivity to, different values of training rates. In
particular,*
- **ML3.4a** *Unless explicitly justified otherwise, ML software
should offer abilities to automatically determine appropriate or
optimal training rates, either as distinct pre-processing
stages, or as implicit stages of model training.*
- **ML3.4b** *ML software which provides default values for
training rates should clearly document anticipated restrictions
of validity of those default values; for example through clear
suggestions that user-determined and -specified values may
generally be necessary or preferable.*
#### 3.1 Control Parameters
Control parameters are considered here to specify how a model is to be
applied to a set of training data. These are generally distinct from
parameters specifying the actual model (such as model architecture).
While we recommend that control parameters be submitted as items of a
single named list, this is neither a firm expectation nor an explicit
part of the current standards.
- **ML3.5** *Parameters controlling optimization algorithms should
minimally include:*
- **ML3.5a** *Specification of the type of algorithm used to
explore the search space (commonly, for example, some kind of
gradient descent algorithm)*
- **ML3.5b** *The kind of loss function used to assess distance
between model estimates and desired output.*
- **ML3.6** *Unless explicitly justified otherwise (for example
because ML software under consideration is an implementation of one
specific algorithm), ML software should:*
- **ML3.6a** *Implement or otherwise permit usage of multiple ways
of exploring search space*
- **ML3.6b** *Implement or otherwise permit usage of multiple loss
functions.*
#### 3.2 CPU and GPU processing
ML software often involves manipulation of large numbers of rectangular
arrays for which graphics processing units (GPUs) are often more
efficient than central processing units (CPUs). ML software thus
commonly offers options to train models using either CPUs or GPUs. While
these standards do not currently suggest any particular design choice in
this regard, we do note the following:
- **ML3.7** *For ML software in which algorithms are coded in C++,
user-controlled use of either CPUs or GPUs (on NVIDIA processors at
least) should be implemented through direct use of
[`libcudacxx`](https://github.com/NVIDIA/libcudacxx).*
This library can be “switched on” through activating a single C++ header
file to switch from CPU to GPU.
### 4 Model Training
Model training is the stage of the ML workflow envisioned here in which
the actual computation is performed by applying a model specified
according to **ML3** to data specified according to **ML1** and **ML2**.
- **ML4.0** *ML software should generally implement a unified
single-function interface to model training, able to receive as
input a model specified according to all preceding standards. In
particular, models with categorically different specifications, such
as different model architectures or optimization algorithms, should
be able to be submitted to the same model training function.*
- **ML4.1** *ML software should at least optionally retain explicit
information on paths taken as an optimizer advances towards minimal
loss. Such information should minimally include:*
- **ML4.1a** *Specification of all model-internal parameters, or
equivalent hashed representation.*
- **ML4.1b** *The value of the loss function at each point*
- **ML4.1c** *Information used to advance to next point, for
example quantification of local gradient.*
- **ML4.2** *The subsequent extraction of information retained
according to the preceding standard should be explicitly documented,
including through example code.*
#### 4.1 Batch Processing
The following standards apply to ML software which implements batch
processing, commonly to train models on data sets too large to be loaded
in their entirety into memory.
- **ML4.3** *All parameters controlling batch processing and
associated terminology should be explicitly documented, and it
should not, for example, be presumed that users will understand the
definition of “epoch” as implemented in any particular ML software.*
According to that standard, it would for example be inappropriate to
have a parameter, `nepochs`, described as “Number of epochs used in
model training”. Rather, the definition and particular implementation of
“epoch” must be explicitly defined.
- **ML4.4** *Explicit guidance should be provided on selection of
appropriate values for parameter controlling batch processing, for
example, on trade-offs between batch sizes and numbers of epochs
(with both terms provided as Control Parameters in accordance with
the preceding standard, **ML3**).*
- **ML4.5** *ML software may optionally include a function to estimate
likely time to train a specified model, through estimating initial
timings from a small sample of the full batch.*
- **ML4.6** *ML software should by default provide explicit
information on the progress of batch jobs (even where those jobs may
be implemented in parallel on GPUs). That information may be
optionally suppressed through additional parameters.*
#### 4.2 Re-sampling
As described at the outset, ML software does not always rely on
pre-specified and categorical distinctions between training and test
data. For example, models may be fit to what is effectively one single
data set in which specified cases or rows are used as training data, and
the remainder as test data. Re-sampling generally refers to the practice
of re-defining categorical distinctions between training and test data.
One training run accordingly connotes training a model on one particular
set of training data and then applying that model to the specified set
of test data. Re-sampling starts that process anew, through constructing
an alternative categorical partition between test and training data.
Even where test and training data are distinguished by more than a
simple data-internal category (such as a labelling column), for example,
by being stored in distinctly-named sub-directories, re-sampling may be
implemented by effectively shuffling data between training and test
sub-directories.
- **ML4.7** *ML software should provide an ability to combine results
from multiple re-sampling iterations using a single parameter
specifying numbers of iterations.*
- **ML4.8** *Absent any additional specification, re-sampling
algorithms should by default partition data according to proportions
of original test and training data.*
- **ML4.8a** *Re-sampling routines of ML software should
nevertheless offer an ability to explicitly control or override
such default proportions of test and training data.*
### 5 Model Output and Performance
Model output is considered here as a stage distinct from model
performance. Model output refers to the end result of model training
(**ML4**), while model performance involves the assessment of a trained
model against a test data set. The present section first describes
standards for model output, which are standards guiding the form of a
model trained according to the preceding standards (**ML4**). Model
Performance is then considered as a separate stage.
#### 5.1 Model Output
- **ML5.0** *The result of applying the training processes described
above should be contained within a single model object returned by
the function defined according to **ML4.0**, above. Even where the
output reflects application to a test data set, the resultant object
need not include any information on model performance (see
**ML5.3**–**ML5.4**, below).*
- **ML5.0a** *That object should either have its own class, or
extend some previously-defined class.*
- **ML5.0b** *That class should have a defined `print` method
which summarises important aspects of the model object,
including but not limited to summaries of input data and
algorithmic control parameters.*
- **ML5.1** *As for the untrained model objects produced according to
the above standards, and in particular as a direct extension of
**ML3.3**, the properties and behaviours of trained models produced
by ML software should be explicitly compared with equivalent objects
produced by other ML software. (Such comparison will generally be
done in terms of comparing model performance, as described in the
following standard **ML5.3**–**ML5.4**).*
- **ML5.2** *The structure and functionality of objects representing
trained ML models should be thoroughly documented. In particular,*
- **ML5.2a** *Either all functionality extending from the class of
model object should be explicitly documented, or a method for
listing or otherwise accessing all associated functionality
explicitly documented and demonstrated in example code.*
- **ML5.2b** *Documentation should include examples of how to save
and re-load trained model objects for their re-use in accordance
with **ML3.1**, above.*
- **ML5.2c** *Where general functions for saving or serializing
objects, such as
[`saveRDS`](https://stat.ethz.ch/R-manual/R-devel/library/base/html/readRDS.html)
are not appropriate for storing local copies of trained models,
an explicit function should be provided for that purpose, and
should be demonstrated with example code.*
The [`R6` system](https://r6.r-lib.org) for representing classes in R is
an example of a system with explicit functionality, all components of
which are accessible by a simple
[`ls()`](https://stat.ethz.ch/R-manual/R-devel/library/base/html/ls.html)
call. Adherence to **ML5.2a** would nevertheless require explicit
description of the ability of
[`ls()`](https://stat.ethz.ch/R-manual/R-devel/library/base/html/ls.html)
to supply a list of all functions associated with an object. The [`mlr`
package](https://github.com/mlr-org/mlr3), for example, uses [`R6`
classes](https://r6.r-lib.org), yet neither explicitly describes the use
of
[`ls()`](https://stat.ethz.ch/R-manual/R-devel/library/base/html/ls.html)
to list all associated functions, nor explicitly lists those functions.
#### 5.2 Model Performance
Model performance refers to the quantitative assessment of a trained
model when applied to a set of test data.
- **ML5.3** *Assessment of model performance should be implemented as
one or more functions distinct from model training.*
- **ML5.4** *Model performance should be able to be assessed according
to a variety of metrics.*
- **ML5.4a** *All model performance metrics represented by
functions internal to a package must be clearly and distinctly
documented.*
- **ML5.4b** *It should be possible to submit custom metrics to a
model assessment function, and the ability to do so should be
clearly documented including through example code.*
The remaining sub-sections specify general standards beyond the
preceding workflow-specific ones.
### 6 Documentation
- **ML6.0** *Descriptions of ML software should make explicit
reference to a workflow which separates training and testing stages,
and which clearly indicates a need for distinct training and test
data sets.*
The following standard applies to packages which are intended or other
able to only encompass a restricted subset of the six primary workflow
steps enumerated at the outset. Envisioned here are packages explicitly
intended to aid one particular aspect of the general workflow envisioned
here, such as implementations of ML optimization functions, or specific
loss measures.
- **ML6.1** *ML software intentionally designed to address only a
restricted subset of the workflow described here should clearly
document how it can be embedded within a typical full ML workflow in
the sense considered here.*
- **ML6.1** *Such demonstrations should include and contrast
embedding within a full workflow using at least two other
packages to implement that workflow.*
### 7 Testing
#### 7.1 Input Data
- **ML7.0** *Test should explicitly confirm partial and
case-insensitive matching of “test”, “train”, and, where applicable,
“validation” data.*
- **ML7.1** *Tests should demonstrate effects of different numeric
scaling of input data (see **ML2.2**).*
- **ML7.2** *For software which imputes missing data, tests should
compare internal imputation with explicit code which directly
implements imputation steps (even where such imputation is a
single-step implemented via some external package). These tests
serve as an explicit reference for how imputation is performed.*
#### 7.2 Model Classes
The following standard applies to models in both untrained and trained
forms, considered to be the respective outputs of the preceding
standards **ML3** and **ML4**.
- **ML7.3** *Where model objects are implemented as distinct classes,
tests should explicitly compare the functionality of these classes
with functionality of equivalent classes for ML model objects from
other packages.*
- **ML7.3a** *These tests should explicitly identify restrictions
on the functionality of model objects in comparison with those
of other packages.*
- **ML7.3b** *These tests should explicitly identify functional
advantages and unique abilities of the model objects in
comparison with those of other packages.*
#### 7.3 Model Training
- **ML7.4** *ML software should explicit document the effects of
different training rates, and in particular should demonstrate
divergence from optima with inappropriate training rates.*
- **ML7.5** *ML software which implements routines to determine
optimal training rates (see **ML3.4**, above) should implement tests
to confirm the optimality of resultant values.*
- **ML7.6** *ML software which implement independent training “epochs”
should demonstrate in tests the effects of lesser versus greater
numbers of epochs.*
- **ML7.7** *ML software should explicitly test different optimization
algorithms, even where software is intended to implement one
specific algorithm.*
- **ML7.8** *ML software should explicitly test different loss
functions, even where software is intended to implement one specific
measure of loss.*
- **ML7.9** *Tests should explicitly compare all possible combinations
in categorical differences in model architecture, such as different
model architectures with same optimization algorithms, same model
architectures with different optimization algorithms, and
differences in both.*
- **ML7.9a** *Such combinations will generally be formed from
multiple categorical factors, for which explicit use of
functions such as
[`expand.grid()`](https://stat.ethz.ch/R-manual/R-devel/library/base/html/expand.grid.html)
is recommended.*
The following example illustrates:
``` r
architechture <- c ("archA", "archB")
optimizers <- c ("optA", "optB", "optC")
cost_fns <- c ("costA", "costB", "costC")
expand.grid (architechture, optimizers, cost_fns)
```
## Var1 Var2 Var3
## 1 archA optA costA
## 2 archB optA costA
## 3 archA optB costA
## 4 archB optB costA
## 5 archA optC costA
## 6 archB optC costA
## 7 archA optA costB
## 8 archB optA costB
## 9 archA optB costB
## 10 archB optB costB
## 11 archA optC costB
## 12 archB optC costB
## 13 archA optA costC
## 14 archB optA costC
## 15 archA optB costC
## 16 archB optB costC
## 17 archA optC costC
## 18 archB optC costC
All possible combinations of these categorical parameters could then be
tested by iterating over the rows of that output.
- **ML7.10** *The successful extraction of information on paths taken
by optimizers (see **ML5.1**, above), should be tested, including
testing the general properties, but not necessarily actual values
of, such data.*
#### 7.4 Model Performance
- **ML7.11** *All performance metrics available for a given class of
trained model should be thoroughly tested and compared.*
- **ML7.11a** *Tests which compare metrics should do so over a
range of inputs (generally implying differently trained models)
to demonstrate relative advantages and disadvantages of
different metrics.*
Sign in with Wallet
Connect another wallet