---
title: Regression Software Standards
tags: statistical-software
robots: noindex, nofollow
---

<!-- Edit the .Rmd not the .md file -->

## Regression and Supervised Learning

This sub-section details standards for Regression and Supervised
Learning Software – referred to from here on for simplicity as
“Regression Software”. Regression Software implements algorithms which
aim to construct or analyse one or more mappings between two defined
data sets (for example, a set of “independent” data, *X*, and a set of
“dependent” data, *Y*). In contrast, the analogous category of
Unsupervised Learning Software aims to construct or analyse one or more
mappings between a defined set of input or independent data, and a
second set of “output” data which are not necessarily known or given
prior to the analysis.

Common purposes of Regression Software are to fit models to estimate
relationships or to make predictions between specified inputs and
outputs. Regression Software includes tools with inferential or
predictive foci, Bayesian, frequentist, or probability-free Machine
Learning (ML) approaches, parametric or or non-parametric approaches,
discrete outputs (such as in classification tasks) or continuous
outputs, and models and algorithms specific to applications or data such
as time series or spatial data. In many cases other standards specific
to these subcategories may apply.

Examples of the diversity of Regression and Unsupervised Learning
software include the following.

1. [`xrnet`](https://joss.theoj.org/papers/10.21105/joss.01761) to
perform “hierarchical regularized regression to incorporate external
data”, where “external data” in this case refers to structured
meta-data as applied to genomic features.
2. [`survPen`](https://joss.theoj.org/papers/10.21105/joss.01434) is,
“an R package for hazard and excess hazard modelling with
multidimensional penalized splines”
3. [`areal`](https://joss.theoj.org/papers/10.21105/joss.01221) is, “an
R package for areal weighted interpolation”.
4. [`ChiRP`](https://joss.theoj.org/papers/10.21105/joss.01287) is a
package for “Chinese Restaurant Process mixtures for regression and
clustering”, which implements a class of non-parametric Bayesian
Monte Carlo models.
5. [`klrfome`](https://joss.theoj.org/papers/10.21105/joss.00722) is a
package for, “kernel logistic regression on focal mean embeddings,”
with a specific and exclusive application to the prediction of
likely archaeological sites.
6. [`gravity`](https://joss.theoj.org/papers/10.21105/joss.01038) is a
package for “estimation methods for gravity models in R,” where
“gravity models” refers to models of spatial interactions between
point locations based on the properties of those locations.
7. [`compboost`](https://joss.theoj.org/papers/10.21105/joss.00967) is
an example of an R package for gradient boosting, which is
inherently a regression-based technique, and so standards for
regression software ought to consider such applications.
8. [`ungroup`](https://joss.theoj.org/papers/10.21105/joss.00937) is,
“an R package for efficient estimation of smooth distributions from
coarsely binned data.” As such, this package is an example of
regression-based software for which the input data are (effectively)
categorical. The package is primarily intended to implement a
particular method for “unbinning” the data, and so represents a
particular class of interpolation methods.
9. [`registr`](https://joss.theoj.org/papers/10.21105/joss.00557) is a
package for “registration for exponential family functional data,”
where registration in this context is effectively an interpolation
method applied within a functional data analysis context.
10. [`ggeffects`](https://joss.theoj.org/papers/10.21105/joss.00772) for
“tidy data frames of marginal effects from regression models.” This
package aims to make statistics quantifying marginal effects readily
understandable, and so implements a standard (tidyverse-based)
methodology for representing and visualising statistics relating to
marginal effects.

Click on the following link to view a demonstration [Application of
Regression and Supervised Learning
Standards](https://hackmd.io/VZ-wgQtZRV2pb-wFZNDM5g).

The following standards are divided among several sub-categories, with
each standard prefixed with “RE”.

### 1 Input data structures and validation

- **RE1.0** *Regression Software should enable models to be specified
via a formula interface, unless reasons for not doing so are
explicitly documented.*
- **RE1.1** *Regression Software should document how formula
interfaces are converted to matrix representations of input data.*

See Max Kuhn’s [RStudio blog
post](https://rviews.rstudio.com/2017/02/01/the-r-formula-method-the-good-parts/)
for examples of how to implement and describe such conversions.

- **RE1.2** *Regression Software should document expected format
(types or classes) for inputting predictor variables, including
descriptions of types or classes which are not accepted.*

Examples documentation addressing this standard include clarifying that
software accepts only numeric inputs in `vector` or `matrix` form, or
that all inputs must be in `data.frame` form with both column and row
names.

- **RE1.3** *Regression Software which passes or otherwise transforms
aspects of input data onto output structures should ensure that
those output structures retain all relevant aspects of input data,
notably including row and column names, and potentially information
from other `attributes()`.*
- **RE1.3a** *Where otherwise relevant information is not
transferred, this should be explicitly documented.*

This standard reflects the common process in regression software of
transforming a rectangular input structure into a modified version which
includes additional columns of model fits or predictions. Software which
constructs such modified versions anew often copies numeric values from
input columns, and may implicitly drop additional information such as
attributes. This standard requires all such information to be retained.

- **RE1.4** *Regression Software should document any assumptions made
with regard to input data; for example distributional assumptions,
or assumptions that predictor data have mean values of zero.
Implications of violations of these assumptions should be both
documented and tested.*

### 2 Pre-processing and Variable Transformation

- **RE2.0** *Regression Software should document any transformations
applied to input data, for example conversion of label-values to
`factor`, and should provide ways to explicitly avoid any default
transformations (with error or warning conditions where
appropriate).*
- **RE2.1** *Regression Software should implement explicit parameters
controlling the processing of missing values, ideally distinguishing
`NA` or `NaN` values from `Inf` values (for example, through use of
`na.omit()` and related functions from the `stats` package).*

Note that fulfilling this standard ensures compliance with all *General
Standard* for missing values (**G2.13**–**G2.16**).

- **RE2.2** *Regression Software should provide different options for
processing missing values in predictor and response data. For
example, it should be possible to fit a model with no missing
predictor data in order to generate values for all associated
response points, even where submitted response values may be
missing.*
- **RE2.3** *Where applicable, Regression Software should enable data
to be centred (for example, through converting to zero-mean
equivalent values; or to z-scores) or offset (for example, to
zero-intercept equivalent values) via additional parameters, with
the effects of any such parameters clearly documented and tested.*
- **RE2.4** *Regression Software should implement pre-processing
routines to identify whether aspects of input data are perfectly
collinear, notably including:*
- **RE2.4a** *Perfect collinearity among predictor variables*
- **RE2.4b** *Perfect collinearity between independent and
dependent variables*

These pre-processing routines should also be tested as described below.

### 3 Algorithms

The following standards apply to the model fitting algorithms of
Regression Software which implement or rely on iterative algorithms
which are expected to converge to generate model statistics. Regression
Software which implements or relies on iterative convergence algorithms
should:

- **RE3.0** *Issue appropriate warnings or other diagnostic messages
for models which fail to converge.*
- **RE3.1** *Enable such messages to be optionally suppressed, yet
should ensure that the resultant model object nevertheless includes
sufficient data to identify lack of convergence.*
- **RE3.2** *Ensure that convergence thresholds have sensible default
values, demonstrated through explicit documentation.*
- **RE3.3** *Allow explicit setting of convergence thresholds, unless
reasons against doing so are explicitly documented.*

### 4 Return Results

- **RE4.0** *Regression Software should return some form of “model”
object, generally through using or modifying existing class
structures for model objects (such as `lm`, `glm`, or model objects
from other packages), or creating a new class of model objects.*
- **RE4.1** *Regression Software may enable an ability to generate a
model object without actually fitting values. This may be useful for
controlling batch processing of computationally intensive fitting
algorithms.*

#### 4.1 Accessor Methods

Regression Software should provide functions to access or extract as
much of the following kinds of model data as possible or practicable.
Access should ideally rely on class-specific methods which extend, or
implement otherwise equivalent versions of, the methods from the `stats`
package which are named in parentheses in each of the following
standards.

Model objects should include, or otherwise enable effectively immediate
access to the following descriptors. It is acknowledged that not all
regression models can sensibly provide access to these descriptors, yet
should include access provisions to all those that are applicable.

- **RE4.2** *Model coefficients (via `coeff()` / `coefficients()`)*
- **RE4.3** *Confidence intervals on those coefficients (via
`confint()`)*
- **RE4.4** *The specification of the model, generally as a formula
(via `formula()`)*
- **RE4.5** *Numbers of observations submitted to model (via
`nobs()`)*
- **RE4.6** *The variance-covariance matrix of the model parameters
(via `vcov()`)*
- **RE4.7** *Where appropriate, convergence statistics*

Note that compliance with **RE4.6** should also heed *General Standard*
**G3.1** in offering user control over covariance algorithms. Regression
Software should further provide simple and direct methods to return or
otherwise access the following form of data and metadata, where the
latter includes information on any transformations which may have been
applied to the data prior to submission to modelling routines.

- **RE4.8** *Response variables, and associated “metadata” where
applicable.*
- **RE4.9** *Modelled values of response variables.*
- **RE4.10** *Model Residuals, including sufficient documentation to
enable interpretation of residuals, and to enable users to submit
residuals to their own tests.*
- **RE4.11** *Goodness-of-fit and other statistics associated such as
effect sizes with model coefficients.*
- **RE4.12** *Where appropriate, functions used to transform input
data, and associated inverse transform functions.*

Regression software may additionally opt to provide simple and direct
methods to return or otherwise access the following:

- **RE4.13** *Predictor variables, and associated “metadata” where
applicable.*

#### 4.2 Prediction, Extrapolation, and Forecasting

Not all regression software is intended to, or can, provide distinct
abilities to extrapolate or forecast. Moreover, identifying cases in
which a regression model is used to extrapolate or forecast may often be
a non-trivial exercise. It may nevertheless be possible, for example
when input data used to construct a model are unidimensional, and data
on which a prediction is to be based extend beyond the range used to
construct the model. Where reasonably unambiguous identification of
extrapolation or forecasting using a model is possible, the following
standards apply:

- **RE4.14** *Where possible, values should also be provided for
extrapolation or forecast *errors*.*
- **RE4.15** *Sufficient documentation and/or testing should be
provided to demonstrate that forecast errors, confidence intervals,
or equivalent values increase with forecast horizons.*

Distinct from extrapolation or forecasting abilities, the following
standard applies to regression software which relies on, or otherwise
provides abilities to process, categorical grouping variables:

- **RE4.16** *Regression Software which models distinct responses for
different categorical groups should include the ability to submit
new groups to `predict()` methods.*

#### 4.3 Reporting Return Results

- **RE4.17** *Model objects returned by Regression Software should
implement or appropriately extend a default `print` method which
provides an on-screen summary of model (input) parameters and
(output) coefficients.*
- **RE4.18** *Regression Software may also implement `summary` methods
for model objects, and in particular should implement distinct
`summary` methods for any cases in which calculation of summary
statistics is computationally non-trivial (for example, for
bootstrapped estimates of confidence intervals).*

### 5 Documentation

Beyond the [*General Standards*](#general-standards) for documentation,
Regression Software should explicitly describe the following aspects,
and ideally provide extended documentation including summary graphical
reports of:

- **RE5.0** *Scaling relationships between sizes of input data
(numbers of observations, with potential extension to numbers of
variables/columns) and speed of algorithm.*

### 6 Visualization

- **RE6.0** *Model objects returned by Regression Software (see*
**RE4***) should have default `plot` methods, either through
explicit implementation, extension of methods for existing model
objects, or through ensuring default methods work appropriately.*
- **RE6.1** *Where the default `plot` method is **NOT** a generic
`plot` method dispatched on the class of return objects (that is,
through an S3-type `plot.<myclass>` function or equivalent), that
method dispatch (or equivalent) should nevertheless exist in order
to explicitly direct users to the appropriate function.*
- **RE6.2** *The default `plot` method should produce a plot of the
`fitted` values of the model, with optional visualisation of
confidence intervals or equivalent.*

The following standard applies only to software fulfilling RE4.14-4.15,
and the conditions described prior to those standards.

- **RE6.3** *Where a model object is used to generate a forecast (for
example, through a `predict()` method), the default `plot` method
should provide clear visual distinction between modelled
(interpolated) and forecast (extrapolated) values.*

### 7 Testing

#### 7.1 Input Data

Tests for Regression Software should include the following conditions
and cases:

- **RE7.0** *Tests with noiseless, exact relationships between
predictor (independent) data.*
- **RE7.0a** In particular, these tests should confirm ability to
reject perfectly noiseless input data.
- **RE7.1** *Tests with noiseless, exact relationships between
predictor (independent) and response (dependent) data.*
- **RE7.1a** *In particular, these tests should confirm that model
fitting is at least as fast or (preferably) faster than testing
with equivalent noisy data (see RE2.4b).*

#### 7.2 Return Results

Tests for Regression Software should

- **RE7.2** Demonstrate that output objects retain aspects of input
data such as row or case names (see **RE1.3**).
- **RE7.3** Demonstrate and test expected behaviour when objects
returned from regression software are submitted to the accessor
methods of **RE4.2**–**RE4.7**.
- **RE7.4** Extending directly from **RE4.15**, where appropriate,
tests should demonstrate and confirm that forecast errors,
confidence intervals, or equivalent values increase with forecast
horizons.