Exploratory Data Analysis (EDA) - Demonstration Application of Standards

This file demonstrates the application of rOpenSci ’s standards for statistical software to three EDA software packages. These applications are not intended to represent or reflect evaluations or assessment of the packages, and particularly not of the extent to which they fail to meet standards. Rather, the demonstrations are intended to highlight aspects of the software which could be productively improved by adhering to the standards, and thereby more generally to demonstrate the general usefulness of these standards in advancing and improving software quality.

Prior to considering these demonstrations, it is recommended to peruse a recent article which appeared in the R Journal entitled, The Landscape of R Packages for Automated Exploratory Data Analysis. A general overview of the packages mentioned in that article is also provided in this github repository (work-in-progres).

SmartEDA

This package conveniently offers a single “master function”, ExpReport, which generates a stand-alone html report containing the output of most of the package’s functions. The current README of the autotest package includes an example of this package which demonstrates a failure of this package to appropriate process or reject rectangular input objects which have list-columns.

1 General Standards

1.1 Documentation

  • G1.0 Primary reference is provided

Statistical Terminology

  • G1.1 Statistical terminology appropriately clarified and defined.

Function-level Documentation

  • G1.2 Software uses roxygen
    • G1.2a internal functions are not appropriately documented

Supplementary Documentation

  • G1.3 No performance claims made so not relevant
  • G1.4 Comparisons with other packages in documentation, yet code not provided.

1.2 Input Structures

Uni-variate (Vector) Input

  • G2.0 No documentation on expected lengths of any input variables
  • G2.1 Types of most inputs clearly documented
  • G2.2 (Not yet implemented in autotest, but will be confirmed there.)
  • G2.3 match.arg() not used for single-valued character inputs, nor is tolower() or equivalent used to avoid sensitivity to case.
  • G2.4 Not applicable
  • G2.5 No mention of how factors are handled

Tabular Input

  • G2.6 All standard input forms accepted
  • G2.7 (Not yet checked)
  • G2.8 Not applicable
  • G2.9 Different classes of tabular input yield consistent results

Missing or Undefined Values

  • G2.10 Missing data appropriately handled
  • G2.11 No user control over missing data needed here
  • G2.12 All missing data appropriately pre-processed
  • G2.13 Undefined values appropriately handled, although no options provided to remove them, even though such options could be useful here.

1.3 Output Structures

  • G4.0 File name specifications (in ExpReport() function) not appropriately parsed, rather simply assumed to be *.html.

1.4 Testing

Test Data Sets

  • G5.0 Tests use data sets provided by other widely-used R packages.
  • G5.1 No data sets created within package, so not applicable

Responses to Unexpected Input

  • G5.2–G5.3 Tests of responses to unexpected input either not given, or do not cover all cases.

Algorithm Tests

  • G5.4–G5.7 Standards for testing statistical algorithms not applicable to EDA software
  • G5.8 No edge condition tests implemented
  • G5.9 No tests of noise susceptibility implemented

Extended tests

  • G5.10–5.12 No extended tests needed, so not applicable

2 EDA Standards

2.1 Documentation Standards

  • EA1.0 No target audience explicitly specified, but abilities of package sufficiently clearly explained to obviate this.
  • EA1.1 No target kinds of data explicitly specified, and they should be.
  • EA1.2 No target questions explicitly identified, but purpose of package is clearly the exploration of associative relationships.
  • EA1.3 The kinds of data each function is intended to accept as input are generally specified.

2.2 Input Data

Index Columns

  • EA2.0-2.3 No table filtering or joining performed, so not relevant

Multi-tabular input

  • EA2.4-2.5 No multi-tabular input possible, so not relevant

Classes and Sub-Classes

  • EA2.6-2.9 Software performs to these standards, as indicated by output of autotest.

2.3 Analytic Algorithms

2.4 Return Results / Output Data

  • EA4.0 integer input types not maintained, rather all values are converted to numeric.
  • EA4.1 Control of numeric precision is explicitly provided throughout, with sensible default values.
  • EA4.2 Default print methods apply to all return objects, but default plot methods either fail (for ExpData), or are not accessible (for ExpNumStat, ExpCTable, and others, which rely on plot.default, and so simply plot a grid of ncol-by-ncol results, often with no labels to enable interpretation).

2.5 Visualization and Summary Output

  • EA5.0-5.1 Explicit graphical output, for example from ExpNumViz and ExpCatViz functions, provides accessible colour schemes, as well as allowing overrides of defaults through additional parameters.

Summary and Screen-based Output

  • EA5.2 Screen-based output has sensibly printed numeric digits, defaulting to 3.
  • EA5.3 Column-based summary statistics do not indicate the storage.mode

General Standards for Visualization (Static and Dynamic)

  • EA5.4 The default plotted output of ExpNumViz includes no scale

2.6 Testing

Return Values

  • EA6.0 Return values from functions generally not tested, including
    • EA6.0a No tests for classes or types of objects
    • EA6.0b No tests for dimensions of tabular objects
    • EA6.0c No tests for column names (or equivalent) of tabular objects
    • EA6.0d No tests for classes or types of all columns contained within data.frame-type tabular objects
    • EA6.0e Tests for equality of single-valued objects do not use a tolerance parameter

Graphical Output

  • EA6.1 properties of graphical output are not tested, merely that output is produced.

insight

3 General Standards

3.1 Documentation

  • G1.0 Primary reference is provided

Statistical Terminology

  • G1.1 Statistical terminology clarified and defined.

Function-level Documentation

  • G1.2 roxygen used to document all functions.
    • G1.2a Internal functions generally not sufficiently documented

Supplementary Documentation

  • G1.3 No performance claims made, so not relevant

3.2 Input Structures

Uni-variate (Vector) Input

  • G2.0 Expected lengths of inputs are generally not documented
  • **G2.1 Expected **data types of vector inputs not documented (for example, the character vectors for component arguments of the find_...() functions).
  • G2.2 There are no checks or restrictions on parameters expected to be univariate.
  • G2.3 No checks implemented for assumed single-valued character input
    • G2.3a match.arg() is not used
    • G2.3b tolower() is not used to avoid sensitivity to case
  • G2.4 Provide appropriate mechanisms to convert between different data types, potentially including:
    • G2.4a There is no explicit conversion to integer via as.integer()
    • G2.4b There is explicit conversion to continuous via as.numeric()
    • G2.4c explicit conversion to character uses as.character()
    • G2.4d explicit conversion to factor uses as.factor()
    • G2.4e explicit conversion from factor uses as...() functions
  • G2.5 No inputs are expected to be of factor type, so not applicable

Tabular Input

  • G2.6–G2.9 No functions accept rectangular input, so not applicable

Missing or Undefined Values

  • G2.10–G2.13 No functions accept data able to contain missing or undefined values, so not applicable

3.3 Output Structures

  • G4.0 No functions use local files, so not appicable

3.4 Testing

Test Data Sets

  • G5.0 Tests use data sets provided by other widely-used R packages.
  • G5.1 No data sets created within package, so not applicable

Responses to Unexpected Input

  • G5.2 Error and warning behaviour of functions is not explicitly tested.
  • G5.3 Absence of missing or undefined values in return objects is not explicitly tested.

Algorithm Tests

  • G5.4–G5.7 Standards for testing statistical algorithms not applicable to EDA software
  • G5.8 Input to software is models not raw data, so edge conditions neither relevant nor applicable.
  • G5.9 Input to software is models not raw data, so noise tests neither relevant nor applicable.

Extended tests

  • G5.10–G5.12 Extended tests neither relevant nor applicable.

4 EDA Standards

4.1 Documentation Standards

  • EA1.0 Target audiences clearly identified
  • EA1.1 The kinds of data the software is capable of analysing clearly identified
  • EA1.2 The kinds of questions the software is intended to help explore are clearly identified
  • EA1.3 The kinds of data each function is intended to accept as input are clearly identified

4.2 Input Data

  • EA2.0–EA2.9 The software does not accept general inputs, so not applicable

4.3 Return Results / Output Data

  • EA4.0 Generally not applicable, as almost all model parameters are numeric, although get_random() does return factor for factor input, so standard met in that regard.
  • EA4.1 Parameters to enable explicit control of numeric precision are not directly implemented, rather the package offers a suite of format_ functions to specify such.
  • EA4.2 Primary routines return objects for which default print and plot methods give sensible results, and also implement additional print_ and format_ methods.

4.4 Visualization and Summary Output

  • EA5.0–EA5.1 There are no plot or other graphical functions.

Summary and Screen-based Output

  • EA5.2 Screen-based output follows getOption("digits"), and so uses default print formatting for numeric types, with no user control possible.
  • EA5.3 Column-based summary statistics do not indicate the storage.mode, class, or equivalent defining attribute of each column.

General Standards for Visualization (Static and Dynamic)

  • EA5.4–EA5.5 There are no visualisations, so not relevant

4.5 Testing

  • EA6.0 Return values from all functions are tested
  • EA6.1 No graphical output produced, so no need to test

naniar

5 General Standards

5.1 Documentation

  • G1.0 Not primary reference given

Statistical Terminology

  • G1.1 Statistical terminology appropriately clarified

Function-level Documentation

  • G1.2 Software should use roxygen to document all functions.
    • G1.2a A scarce few internal functions are not be documented in roxygen format

Supplementary Documentation

  • G1.3 No performance claims make so not relevant.

5.2 Input Structures

Uni-variate (Vector) Input

  • G2.0 There is no documentation of expectations on lengths of inputs
  • G2.1 Provide explicit secondary documentation of expectations on data types of all vector inputs (see the above list).
  • G2.2 Lengths of parameters expected to be univariate are not checked, and lead to uninformative errors (for example, missing parameter of add_any_miss() function).
  • G2.3 No single-valued character parameters used, so not applicable.
  • G2.4 Provide appropriate mechanisms to convert between different data types, potentially including:
    • G2.4a explicit conversion to integer uses as.integer()
    • G2.4b explicit conversion to continuous uses as.numeric()
    • G2.4c There is no explicit conversion to character, so not applicable paste or paste0)
    • G2.4d There is no explicit conversion to factor, so not applicable
    • G2.4e There is no explicit conversion from factor, so not applicable
  • G2.5 No inputs expected to be of factor type, so not applicable.

Tabular Input

  • G2.6 naniar does not accept matrix/array input, even though it easily could.
  • G2.7 There is no initial pre-processing to ensure that all other sub-functions of a package receive inputs of a single defined type.
  • G2.8 There is no type conversion, so no diagnostic messages for type conversion are issued.
  • G2.9 Extraction/filtering of single columns from tabular inputs behaves appropriately regardless of input class.

Missing or Undefined Values

  • G2.10 Checks for missing data are a core part of every routine
  • G2.11 Most functions provide multiple, custom options for handling missing
  • G2.12 No functions assume non-missingness
  • G2.13 No functions appropriately handle undefined values other than NA. NaN is treated exactly as NA, and Inf is simply ignored.

5.3 Output Structures

  • G4.0 There are no outputs written to local files, so not applicable

5.4 Testing

Test Data Sets

  • G5.0 Tests use data sets provided by other widely-used R packages.
  • G5.1 No data sets created within package, so not applicable exported so that users can confirm tests and run examples.

Responses to Unexpected Input

  • G5.2 Error and warning behaviour not tested for all functions
  • G5.3 Absence of missing or undefined values is tested

Algorithm Tests

  • G5.4–G5.8 Standards for testing statistical algorithms not applicable to EDA software
  • G5.8 Edge conditions tested appropriately
  • G5.9 Test of noise susceptibility not applicable

Extended tests

  • G5.10–G5.12 Extended tests neither relevant nor applicable.

6 EDA Standards

6.1 Documentation Standards

  • EA1.0 The software does not identify any target audiences for whom it is intended
  • EA1.1 The software does not explicitly identify the kinds of data it is capable of analysing, rather refers to all inputs as (variously and either), “data frame” or “data.frame”, even though any rectangular inputs are accepted.
  • EA1.2 The software clearly identifies the kinds of questions it is intended to help explore
  • EA1.3 The software does not identify the kinds of data each function is intended to accept as input (see EA1.1, above).

6.2 Input Data

Index Columns

  • EA2.0–2.2 Software does not rely on index columns
  • EA2.3 Table join operations are not based on any assumed variable or column names, rather are only performed on inputs generated internally within the package to have standard format.

Multi-tabular input

  • EA2.4–EA2.5 Package does not use or admit multi-tabular input

Classes and Sub-Classes

  • EA2.6 Routines appropriately process vector input of custom classes
  • EA2.7 Routines appropriately process vector data regardless of additional attributes
  • EA2.8 Routines appropriately process rectangular input of custom classes
  • EA2.9 Routines accept and appropriately process rectangular input with columns of custom sub-classes including additional attributes.

6.3 Analytic Algorithms

6.4 Return Results / Output Data

  • EA4.0 Software does not ensure all return results have types which are consistent with input types, rather returns all results as tibble objects regardless of class of input.
  • EA4.1 Universal use of tibble classes ensures explicit control of numeric precision
  • EA4.2 Universal use of tibble classes ensures default print and plot methods give sensible results.

6.5 Visualization and Summary Output

  • EA5.0 Graphical presentation is as accessible as possible or practicable.
  • EA5.1 Typefaces appear to consider accessibility

Summary and Screen-based Output

  • EA5.2 Screen-based output does not rely on default print formatting of numeric types, rather relies on print.tibble throughout.
  • EA5.3 Column-based summary statistics always indicates the storage.mode via tibble.

General Standards for Visualization (Static and Dynamic)

  • EA5.4 Visualisations do not include units on all axes, but do use ggplot2 to produce sensibly rounded values.

Dynamic Visualization

  • EA5.5 There are no routines for dynamic visualization.

6.6 Testing

Return Values

  • EA6.0 Return values from all functions are tested

Graphical Output

  • EA6.1 The properties of graphical output are explicitly tested
Select a repo