---
title: Exploratory Data Analysis Software Standards
tags: statistical-software
robots: noindex, nofollow
---

<!-- Edit the .Rmd not the .md file -->

## Exploratory Data Analysis

Exploration is a part of all data analyses, and Exploratory Data
Analysis (EDA) is not something that is entered into and exited from at
some point prior to “real” analysis. Exploratory Analyses are also not
strictly limited to *Data*, but may extend to exploration of *Models* of
those data. The category could thus equally be termed, “*Exploratory
Data and Model Analysis*”, yet we opt to utilise the standard acronym of
EDA in this document.

EDA is nevertheless somewhat different to many other categories included
within rOpenSci’s program for peer-reviewing statistical software.
Primary differences include:

- EDA software often has a strong focus upon visualization, which is a
category which we have otherwise explicitly excluded from the scope
of the project at the present stage.
- The assessment of EDA software requires addressing more general
questions than software in most other categories, notably including
the important question of intended audience(s).

Examples of EDA software include:

1. A package rejected by rOpenSci as out-of-scope,
[`gtsummary`](https://github.com/ddsjoberg/gtsummary), which
provides, “Presentation-ready data summary and analytic result
tables.” Other examples include:
2. The [`smartEDA` package](https://github.com/daya6489/SmartEDA) (with
accompanying [JOSS
paper](https://joss.theoj.org/papers/10.21105/joss.01509)) “for
automated exploratory data analysis”. The package, “automatically
selects the variables and performs the related descriptive
statistics. Moreover, it also analyzes the information value, the
weight of evidence, custom tables, summary statistics, and performs
graphical techniques for both numeric and categorical variables.”
This package is potentially as much a workflow package as it is a
statistical reporting package, and illustrates the ambiguity between
these two categories.
3. The [`modeLLtest`
package](https://github.com/ShanaScogin/modeLLtest) (with
accompanying [JOSS
paper](https://joss.theoj.org/papers/10.21105/joss.01542)) is “An R
Package for Unbiased Model Comparison using Cross Validation.” Its
main functionality allows different statistical models to be
compared, likely implying that this represents a kind of meta
package.
4. The [`insight` package](https://github.com/easystats/insight) (with
accompanying [JOSS
paper](https://joss.theoj.org/papers/10.21105/joss.01412) provides
“a unified interface to access information from model objects in R,”
with a strong focus on unified and consistent reporting of
statistical results.
5. The [`arviz` software for
python](https://github.com/arviz-devs/arviz) (with accompanying
[JOSS paper](https://joss.theoj.org/papers/10.21105/joss.01143)
provides “a unified library for exploratory analysis of Bayesian
models in Python.”
6. The [`iRF` package](https://github.com/sumbose/iRF) (with
accompanying [JOSS
paper](https://joss.theoj.org/papers/10.21105/joss.01077) enables
“extracting interactions from random forests”, yet also focusses
primarily on enabling interpretation of random forests through
reporting on interaction terms.

Click on the following link to view a demonstration [Application of
Exploratory Data Analysis
Standards](https://hackmd.io/K8F1RIhdQeuZFqMnzdqNVw).

Reflecting these considerations, the following standards are somewhat
differently structured than equivalent standards developed to date for
other categories, particularly through being more qualitative and
abstract. In particular, while documentation is an important component
of standards for all categories, clear and instructive documentation is
of paramount importance for EDA Software, and so warrants its own
sub-section within this document.

### 1 Documentation Standards

The following refer to *Primary Documentation*, implying in main package
`README` or vignette(s), and *Secondary Documentation*, implying
function-level documentation.

The *Primary Documentation* (`README` and/or vignette(s)) of EDA
software should:

- **EA1.0** *Identify one or more target audiences for whom the
software is intended*
- **EA1.1** *Identify the kinds of data the software is capable of
analysing (see *Kinds of Data\* below).\*
- **EA1.2** *Identify the kinds of questions the software is intended
to help explore.*

Important distinctions between kinds of questions include whether they
are inferential, predictive, associative, causal, or representative of
other modes of statistical enquiry. The *Secondary Documentation*
(within individual functions) of EDA software should:

- **EA1.3** *Identify the kinds of data each function is intended to
accept as input*

### 2 Input Data

A further primary difference of EDA software from that of our other
categories is that input data for statistical software may be generally
presumed of one or more specific types, whereas EDA software often
accepts data of more general and varied types. EDA software should aim
to accept and appropriately transform as many diverse kinds of input
data as possible, through addressing the following standards, considered
in terms of the two cases of input data in uni- and multi-variate form.
All of the general standards for kinds of input (G2.0 - G2.12) apply to
input data for EDA Software.

#### 2.1 Index Columns

The following standards refer to an *index column*, which is understood
to imply an explicitly named or identified column which can be used to
provide a unique index index into any and all rows of that table. Index
columns ensure the universal applicability of standard table join
operations, such as those implemented via the [`dplyr`
package](https://dplyr.tidyverse.org).

- **EA2.0** *EDA Software which accepts standard tabular data and
implements or relies upon extensive table filter and join operations
should utilise an **index column** system*
- **EA2.1** *All values in an index column must be unique, and this
uniqueness should be affirmed as a pre-processing step for all input
data.*
- **EA2.2** *Index columns should be explicitly identified, either:*
- **EA2.2a** *by using an appropriate class system, or*
- **EA2.2b** *through setting an `attribute` on a table, `x`, of
`attr(x, "index") <- <index_col_name>`.*

For EDA software which either implements custom classes or explicitly
sets attributes specifying index columns, these attributes should be
used as the basis of all table join operations, and in particular:

- **EA2.3** *Table join operations should not be based on any assumed
variable or column names*

#### 2.2 Multi-tabular input

EDA software designed to accept multi-tabular input should:

- **EA2.4** *Use and demand an explicit class system for such input
(for example, via the [`DM`
package](https://github.com/krlmlr/dm)).*
- **EA2.5** *Ensure all individual tables follow the above standards
for Index Columns*

#### 2.3 Classes and Sub-Classes

*Classes* are understood here to be the classes define single input
objects, while *Sub-Classes* refer to the class definitions of
components of input objects (for example, of columns of an input
`data.frame`). EDA software which is intended to receive input in
general vector formats (see *Uni-variate Input* section of [*General
Standards*](#general-standards)) should ensure that it complies with
**G2.**, so that vector input is appropriately processed regardless of
input class. An additional standard for EDA software is that,

- **EA2.6** *Routines should appropriately process vector data
regardless of additional attributes*

The following code illustrates some ways by which “metadata” defining
classes and additional attributes associated with a standard vector
object may by modified.

x <- 1:10
class (x) <- "notvector"
attr (x, "extra_attribute") <- "another attribute"
attr (x, "vector attribute") <- runif (5)
attributes (x)
#> $class
#> [1] "notvector"
#> 
#> $extra_attribute
#> [1] "another attribute"
#> 
#> $`vector attribute`
#> [1] 0.03521663 0.49418081 0.60129563 0.75804346 0.16073301

All statistical software should appropriately deal with such input data,
as exemplified by the `storage.mode()`, `length()`, and `sum()`
functions of the `base` package, which return the appropriate values
regardless of redefinition of class or additional attributes.

storage.mode (x)
#> [1] "integer"
length (x)
#> [1] 10
sum (x)
#> [1] 55
storage.mode (sum (x))
#> [1] "integer"

Tabular inputs in `data.frame` class may contain columns which are
themselves defined by custom classes, and which possess additional
attributes. The ability of software to accept such inputs is covered by
the *Tabular Input* section of the [*General
Standards*](#general-standards).

### 3 Analytic Algorithms

EDA software will generally not directly implement what might be
considered as statistical algorithms in their own right. Where
algorithms are implemented, the following standards apply.

- **EA3.0** *The algorithmic components of EDA Software should enable
automated extraction and/or reporting of statistics as some
sufficiently “meta” level (such as variable or model selection), for
which previous or reference implementations require manual
intervention.*
- **EA3.1** *EDA software should enable standardised comparison of
inputs, processes, models, or outputs which previous or reference
implementations otherwise only enable in some comparably
unstandardised form.*

Both of these standards also relate to the following standards for
output values, visualisation, and summary output.

### 4 Return Results / Output Data

- **EA4.0** *EDA Software should ensure all return results have types
which are consistent with input types.*

Examples of such compliance include ensuring that `sum`, `min`, or `max`
values applied to `integer`-type vectors return `integer` values.

- **EA4.1** *EDA Software should implement parameters to enable
explicit control of numeric precision*
- **EA4.2** *The primary routines of EDA Software should return
objects for which default `print` and `plot` methods give sensible
results. Default `summary` methods may also be implemented.*

### 5 Visualization and Summary Output

Visualization commonly represents one of the primary functions of EDA
Software, and thus visualization output is given greater consideration
in this category than in other categories in which visualization may
nevertheless play an important role. In particular, one component of
this sub-category is *Summary Output*, taken to refer to all forms of
screen-based output beyond conventional graphical output, including
tabular and other text-based forms. Standards for visualization itself
are considered in the two primary sub-categories of static and dynamic
visualization, where the latter includes interactive visualization.

Prior to these individual sub-categories, we consider a few standards
applicable to visualization in general, whether static or dynamic.

- **EA5.0** *Graphical presentation in EDA software should be as
accessible as possible or practicable. In particular, EDA software
should consider accessibility in terms of:*
- **EA5.0a** *Typeface sizes, which should default to sizes which
explicitly enhance accessibility*
- **EA5.0b** *Default colour schemes, which should be carefully
constructed to ensure accessibility.*
- **EA5.1** *Any explicit specifications of typefaces which override
default values provided through other packages (including the
`graphics` package) should consider accessibility*

#### 5.1 Summary and Screen-based Output

- **EA5.2** *Screen-based output should never rely on default print
formatting of `numeric` types, rather should also use some version
of `round(., digits)`, `formatC`, `sprintf`, or similar functions
for numeric formatting according the parameter described in*
**EA4.1**.
- **EA5.3** *Column-based summary statistics should always indicate
the `storage.mode`, `class`, or equivalent defining attribute of
each column.*

An example of compliance with the latter standard is the `print.tibble`
method of the [`tibble` package](https://tibble.tidyverse.org).

#### 5.2 General Standards for Visualization (Static and Dynamic)

- **EA5.4** *All visualisations should ensure values are rounded
sensibly (for example, via `pretty()` function).*
- **EA5.5** *All visualisations should include units on all axes where
such are specified or otherwise obtainable from input data or other
routines.*

#### 5.3 Dynamic Visualization

Dynamic visualization routines are commonly implemented as interfaces to
`javascript` routines. Unless routines have been explicitly developed as
an internal part of an R package, standards shall not be considered to
apply to the code itself, rather only to decisions present as
user-controlled parameters exposed within the R environment. That said,
one standard may nevertheless be applied, which aims to maximise
inter-operability between packages.

- **EA5.6** *Any packages which internally bundle libraries used for
dynamic visualization and which are also bundled in other,
pre-existing R packages, should explain the necessity and advantage
of re-bundling that library.*

### 6 Testing

#### 6.1 Return Values

- **EA6.0** *Return values from all functions should be tested,
including tests for the following characteristics:*
- **EA6.0a** *Classes and types of objects*
- **EA6.0b** *Dimensions of tabular objects*
- **EA6.0c** *Column names (or equivalent) of tabular objects*
- **EA6.0d** *Classes or types of all columns contained within
`data.frame`-type tabular objects *
- **EA6.0e** *Values of single-valued objects; for `numeric`
values either using `testthat::expect_equal()` or equivalent
with a defined value for the `tolerance` parameter, or using
`round(..., digits = x)` with some defined value of `x` prior to
testing equality.*

#### 6.2 Graphical Output

- **EA6.1** *The properties of graphical output from EDA software
should be explicitly tested, for example via the [`vdiffr`
package](https://github.com/r-lib/vdiffr) or equivalent.*

Tests for graphical output are frequently only run as part of an
extended test suite.