Unsupervised Learning Demonstrations

This file demonstrates the application of rOpenSci ’s standards for statistical software to one Unsupervised Learning software package. These applications are not intended to represent or reflect evaluations or assessment of the packages, and particularly not of the extent to which they fail to meet standards. Rather, the demonstrations are intended to highlight aspects of the software which could be productively improved by adhering to the standards, and thereby more generally to demonstrate the general usefulness of these standards in advancing and improving software quality.

mbkmeans

1 General Standards

• G1.0–1.1 Clear performance claims are made in the associated publication, yet the software includes no code able to reproduce these results, nor to compare with alternative implementations.
• G2.0 There is no documentation of any expectations on lengths of inputs.
• G2.1 There is no secondary documentation of expectations on data types of vector inputs
• G2.2 Multivariate input to parameters expected to be univariate is generally appropriately prohibited
• G2.3 For univariate character input:
  • G2.3a match.arg() is not used, and would yield more sensible errors
  • G2.3b tolower() or equivalent is not used, and would yield more sensible errors
• G2.4 Provide appropriate mechanisms to convert between different data types, potentially including:
  • G2.4a There is no explicit conversion to integer via as.integer(), and should be (for example for clusters, which is ultimately implicitly converted by being passed as an assumed C++ <int>).
  • G2.4b Nor is there any explicit conversion via as.numeric()
  • G2.4c Nor is there any explicit conversion via as.character() (for example for reduceMethod).
  • G2.4d–G2.4e Conversion to/from factor not used, so not applicable
• G2.5 No inputs expected to be of factor type, so not applicable
• G2.6 Only input of specified classes permitted, with appropriate class checks impelemented.
• G2.7 Software does not provide appropriate routines to convert main input to standard form. For example, all input matrices are transposed without pre-processing checks for row-column orientation.
• G2.8 Input objects generally include a large amount of meta-data, most of which is lost. In this context, such silent discarding nevertheless seems appropriate.
• G2.9 Primary inputs derive from matrix, and so do not use list columns, therefore not applicable.
• G2.10 Checks for missing data are not implemented as pre-processing steps, rather data is passed to main routines (the C++ mini_batch routine), where uninformative errors are generated.
• G2.11 No options are provided for users to specify how to handle missing data.
• G2.12 Functions assume non-missingness, and generate unhelpful error messages when missing data are submitted.
• G2.13 There are neither options, nor pre-processing checks, for undefined values, rather data is again passed to C++ routines, resulting in uninformative error messages.
• G4.0 The package itself relies on external packages to produce local files, so not applicable
• G5.0 Tests use standard data sets with known properties (notably iris).
• G5.1 No data sets created within, and used to test, package, so not applicable.
• G5.2 Tests do not exist for appropriate error and warning behaviour of all functions
• G5.3 The absence of missing or undefined values in return objects is not explicitly tested.
• G5.4 Correctness tests appropriately implemented
  • G5.4a No tests again alternative implementations, yet such tests could be plausibly included.
  • G5.4b Tests against previous implementations not applicable.
  • G5.4c Use of stored values from published paper outputs not applicable
• G5.5 Correctness tests are run with a fixed random seed
• G5.6 Parameter recovery tests appropriately impelemented.
  • G5.6a Parameter recovery tests use defined tolerance.
  • G5.6b Parameter recovery tests are not run with multiple random seeds
• G5.7 There are no algorithm performance tests (and I am not sure these would be relevant here?)
• G5.8 There are no edge condition tests, and there should be
  • G5.8a No tests for zero-length data
  • G5.8b No tests for data of unsupported types
  • G5.8c No tests for data with all-NA fields or columns or all identical fields or columns
  • G5.8d No tests for data outside the scope of the algorithm
• G5.9 There are no noise susceptibility tests
  • G5.9a There are no tests that adding trivial noise does not meaningfully change results, and there should be.
  • G5.9b There are no tests that running under different random seeds or initial conditions does not meaningfully change results, and there should be
• G5.10–G5.12 There are no extended tests, so not applicable.

2 Unsupervised Learning Standards

• UL1.0 Although there is explicit documentation of expected format for input data, there are no descriptions of types or classes which are not accepted, nor sufficiently clarity (for example, in the function examples) on exactly what is accepted.
• UL1.1 Sub-routines assert that all input data is of the expected form, with informative error messages issued when incompatible data are submitted.
• UL1.2–3 Input inherits from matrix only, so row or column names not used, and these standards not applicable.
• UL1.4 There is no explicit documentation on whether input data may include missing values.
• UL1.5 Functions do not provide informative error messages when data with missing values are submitted.
• UL1.6 There is no documentation of assumptions made with regard to input data, and there likely should be (for mini_batch, for example).
  • UL1.6a Software responds qualitatively differently to input data which has components on markedly different scales, yet this is not documented (for mini_batch, for example).
  • UL1.6b There are no illustrations or contrasts of the consequences of submitted scaled versus unscaled data.
• UL2.0 Routines are likely to give unreliable or irreproducible results in response to violations of assumptions regarding input data, yet there are no pre-processing checks for such.
• UL2.1 Transformations are applied to input data without documentation or ways of avoiding (notably transposing the input to mbkmeans()).
• UL2.2 Missing values not accepted in input data, so not applicable.
• UL2.3 There are no pre-processing routines to identify whether aspects of input data are perfectly collinear.
• UL3.1 No labels applied to input data, so not applicable.
• UL3.2 There is no labelling of dimensions or groups, so not applicable.
• UL3.3 Input data does not generally include labels, yet there is no additional parameter to enable cases to be labelled, and there should be.
• UL3.4 Prediction of the properties of additional new data neither possible nor applicable to this software.
• UL3.5 There is no quantitative information on intra-group variances or equivalent, yet this could be provided.
• UL4.0 Return values are not “model” objects, rather simple lists not immediately able to be submitted to any further functions.
• UL4.1 Ability to generate a model object without actually fitting values arguably not applicable here.
• UL4.2 Return objects neither include, nor enable immediate extraction of, parameters used to control the algorithm used.
• UL4.2 There is no default print method for objects returned from main functions.
  • UL4.2a In the absence of the above, the default print method fails to ensure only a restricted number of rows of any result matrices or equivalent are printed to the screen.
• UL4.3 The default summary.list methods for return objects are arguably sufficient here (although a non-default method would enable more informative summary data to be generated).
• UL6.0–6.2 There are no default plot methods.

3 Summary

4 autotest output

Current autotest only produces output where issues arises. This is a relatively very clean response, none of which ought be considered particularly important, except perhaps for the notes about missing documentation on permitted parameter ranges.

• Parameter clusters responds to integer values in [1, 110]
• Warning: ✖ Parameter range for clusters is NOT documented
• Parameter batch_size responds to integer values in [1, 110]
• Warning: ✖ Parameter range for batch_size is NOT documented
• Parameter max_iters permits unrestricted integer inputs
• Parameter initializer of function mini_batch is assumed to a single character, but is case dependent
• Parameter compute_labels of function mini_batch is assumed to be logical, but responds to general integer values.
• Parameter calc_wcss of function mini_batch is assumed to be logical, but responds to general integer values.
• Parameter early_stop_iter responds to integer values in [1, 28]
• Warning: ✖ Parameter range for early_stop_iter is NOT documented
• Parameter verbose of function mini_batch is assumed to be logical, but responds to general integer values.