---
title: unsupervised-demos
tags: statistical-software-demos, statistical-software
robots: noindex, nofollow
---

Unsupervised Learning Demonstrations
====================================

This file demonstrates the application of
[rOpenSci](https://ropensci.org) ’s [standards for statistical
software](https://ropenscilabs.github.io/statistical-software-review-book/standards.html)
to one [Unsupervised Learning
software](https://ropenscilabs.github.io/statistical-software-review-book/standards.html#dimensionality-reduction-clustering-and-unsupervised-learning)
package. These applications are not intended to represent or reflect
evaluations or assessment of the packages, and particularly not of the
extent to which they fail to meet standards. Rather, the demonstrations
are intended to highlight aspects of the software which could be
productively improved by adhering to the standards, and thereby more
generally to demonstrate the general usefulness of these standards in
advancing and improving software quality.

[`mbkmeans`](https://github.com/drisso/mbkmeans)
------------------------------------------------

### 1 General Standards

- [ ] **G1.0–1.1** Clear performance claims are made in the associated
publication, yet the software includes no code able to reproduce
these results, nor to compare with alternative implementations.
- [ ] **G2.0** There is no documentation of any expectations on
lengths of inputs.
- [ ] **G2.1** There is no secondary documentation of expectations on
*data types* of vector inputs
- [x] **G2.2** Multivariate input to parameters expected to be
univariate is generally appropriately prohibited
- [ ] **G2.3** For univariate character input:
- [ ] **G2.3a** `match.arg()` is not used, and would yield more
sensible errors
- [ ] **G2.3b** `tolower()` or equivalent is not used, and would
yield more sensible errors
- [ ] **G2.4** Provide appropriate mechanisms to convert between
different data types, potentially including:
- [ ] **G2.4a** There is no explicit conversion to `integer` via
`as.integer()`, and should be (for example for `clusters`, which
is ultimately *implicitly* converted by being passed as an
assumed [C++
`<int>`](https://github.com/drisso/mbkmeans/blob/master/src/mini_batch.cpp#L394)).
- [ ] **G2.4b** Nor is there any explicit conversion via
`as.numeric()`
- [ ] **G2.4c** Nor is there any explicit conversion via
`as.character()` (for example for `reduceMethod`).
- [x] **G2.4d**–**G2.4e** Conversion to/from factor not used, so
not applicable
- [x] **G2.5** No inputs expected to be of `factor` type, so not
applicable
- [x] **G2.6** Only input of specified classes permitted, with
[appropriate class checks
impelemented](https://github.com/drisso/mbkmeans/blob/master/R/kmeans.R#L169-L170).
- [ ] **G2.7** Software does not provide appropriate routines to
convert main input to standard form. For example, all input matrices
are
[transposed](https://github.com/drisso/mbkmeans/blob/master/R/kmeans.R#L174)
without pre-processing checks for row-column orientation.
- [ ] **G2.8** Input objects generally include a large amount of
meta-data, most of which is lost. In this context, such silent
discarding nevertheless seems appropriate.
- [x] **G2.9** Primary inputs derive from `matrix`, and so do not use
list columns, therefore not applicable.
- [ ] **G2.10** Checks for missing data are not implemented as
pre-processing steps, rather data is passed to main routines (the
C++ `mini_batch` routine), where uninformative errors are generated.
- [ ] **G2.11** No options are provided for users to specify how to
handle missing data.
- [ ] **G2.12** Functions assume non-missingness, and generate
unhelpful error messages when missing data are submitted.
- [ ] **G2.13** There are neither options, nor pre-processing checks,
for undefined values, rather data is again passed to C++ routines,
resulting in uninformative error messages.
- [x] **G4.0** The package itself relies on external packages to
produce local files, so not applicable
- [x] **G5.0** Tests use standard data sets with known properties
(notably `iris`).
- [x] **G5.1** No data sets created within, and used to test, package,
so not applicable.
- [ ] **G5.2** Tests do not exist for appropriate error and warning
behaviour of all functions
- [ ] **G5.3** The absence of missing or undefined values in return
objects is not explicitly tested.
- [ ] **G5.4** **Correctness tests** appropriately implemented
- [ ] **G5.4a** No tests again alternative implementations, yet
such tests could be plausibly included.
- [x] **G5.4b** Tests against previous implementations not
applicable.
- [x] **G5.4c** Use of stored values from published paper outputs
not applicable
- [x] **G5.5** Correctness tests are run with a fixed random seed
- [x] **G5.6** **Parameter recovery tests** appropriately
impelemented.
- [x] **G5.6a** Parameter recovery tests use defined tolerance.
- [ ] **G5.6b** Parameter recovery tests are not run with multiple
random seeds
- [ ] **G5.7** There are no algorithm performance tests (and I am not
sure these would be relevant here?)
- [ ] **G5.8** There are no edge condition tests, and there should be
- [ ] **G5.8a** No tests for zero-length data
- [ ] **G5.8b** No tests for data of unsupported types
- [ ] **G5.8c** No tests for data with all-`NA` fields or columns
or all identical fields or columns
- [ ] **G5.8d** No tests for data outside the scope of the
algorithm
- [ ] **G5.9** There are no noise susceptibility tests
- [ ] **G5.9a** There are no tests that adding trivial noise does
not meaningfully change results, and there should be.
- [ ] **G5.9b** There are no tests that running under different
random seeds or initial conditions does not meaningfully change
results, and there should be
- [x] **G5.10**–**G5.12** There are no extended tests, so not
applicable.

### 2 Unsupervised Learning Standards

- [ ] UL1.0 Although there is explicit documentation of expected
format for input data, there are no descriptions of types or classes
which are not accepted, nor sufficiently clarity (for example, in
the function examples) on exactly what is accepted.
- [x] UL1.1 Sub-routines assert that all input data is of the expected
form, with informative error messages issued when incompatible data
are submitted.
- [x] UL1.2–3 Input inherits from `matrix` only, so row or column
names not used, and these standards not applicable.
- [ ] UL1.4 There is no explicit documentation on whether input data
may include missing values.
- [ ] UL1.5 Functions do not provide informative error messages when
data with missing values are submitted.
- [ ] UL1.6 There is no documentation of assumptions made with regard
to input data, and there likely should be (for `mini_batch`, for
example).
- [ ] UL1.6a Software responds qualitatively differently to input
data which has components on markedly different scales, yet this
is not documented (for `mini_batch`, for example).
- [ ] UL1.6b There are no illustrations or contrasts of the
consequences of submitted scaled versus unscaled data.
- [ ] UL2.0 Routines are likely to give unreliable or irreproducible
results in response to violations of assumptions regarding input
data, yet there are no pre-processing checks for such.
- [ ] UL2.1 Transformations are applied to input data without
documentation or ways of avoiding (notably transposing the input to
`mbkmeans()`).
- [x] UL2.2 Missing values not accepted in input data, so not
applicable.
- [ ] UL2.3 There are no pre-processing routines to identify whether
aspects of input data are perfectly collinear.
- [x] UL3.1 No labels applied to input data, so not applicable.
- [x] UL3.2 There is no labelling of dimensions or groups, so not
applicable.
- [ ] UL3.3 Input data does not generally include labels, yet there is
no additional parameter to enable cases to be labelled, and there
should be.
- [x] UL3.4 Prediction of the properties of additional new data
neither possible nor applicable to this software.
- [ ] UL3.5 There is no quantitative information on intra-group
variances or equivalent, yet this could be provided.
- [ ] UL4.0 Return values are not “model” objects, rather simple lists
not immediately able to be submitted to any further functions.
- [x] UL4.1 Ability to generate a model object without actually
fitting values arguably not applicable here.
- [ ] UL4.2 Return objects neither include, nor enable immediate
extraction of, parameters used to control the algorithm used.
- [ ] UL4.2 There is no default `print` method for objects returned
from main functions.
- [ ] UL4.2a In the absence of the above, the default `print`
method fails to ensure only a restricted number of rows of any
result matrices or equivalent are printed to the screen.
- [x] UL4.3 The default `summary.list` methods for return objects are
arguably sufficient here (although a non-default method would enable
more informative `summary` data to be generated).
- [ ] UL6.0–6.2 There are no default `plot` methods.

### 3 Summary

| Standards | Total Number | Pass | Not Applicable | Fail |
|-----------|--------------|------|----------------|------|
| General | 28 | 5 | 7 | 16 |
| UL | 24 | 2 | 6 | 16 |

### 4 [`autotest`](https://github.com/mpadge/autotest) output

Current `autotest` only produces output where issues arises. This is a
relatively very clean response, none of which ought be considered
particularly important, except perhaps for the notes about missing
documentation on permitted parameter ranges.

- Parameter `clusters` responds to integer values in \[1, 110\]
- Warning: ✖ Parameter range for clusters is NOT documented
- Parameter `batch_size` responds to integer values in \[1, 110\]
- Warning: ✖ Parameter range for batch\_size is NOT documented
- Parameter `max_iters` permits unrestricted integer inputs
- Parameter `initializer` of function `mini_batch` is assumed to a
single character, but is case dependent
- Parameter `compute_labels` of function `mini_batch` is assumed to be
logical, but responds to general integer values.
- Parameter `calc_wcss` of function `mini_batch` is assumed to be
logical, but responds to general integer values.
- Parameter `early_stop_iter` responds to integer values in \[1, 28\]
- Warning: ✖ Parameter range for `early_stop_iter` is NOT documented
- Parameter verbose of function `mini_batch` is assumed to be logical,
but responds to general integer values.