NMinD Certification Checklists

Three tiers of increasing rigidity to the NMinD standard.

Releasing software via this certification process will let other user know you followed coding practices that will ensure transparency, reproducibility and collaboration. This way, your code will be used and cited very often!

The three tier model increases on the level of detail of user's documentation and developer's documentation.

---

Bronze Certification

This tool exists and is reasonably useable by (at least) advanced users, with minimal useability for basic users.

TODO: Insert example Bronze cookie-cutter here.

Documentation

• Brief description of what program does is listed.
• An example of its usage is shown.
  • typical usage
• Description of required input parameters.
  • Reasonable description (i.e. "Nifti of brain mask in MNI" vs. "An image file")
• Description of output(s).

Infrastructure

• Package is under version control.
• Readme is present.
• License is present.
• Issues tracking is enabled (either through GitHub or external site).
• Contributing guidelines (Intent of not maintenance?)
  • Adding NMinD template for users with important questions
• [Digital Object Identifier] - (Zenodo)

Code

• Installation instructions available.
• Dependencies listed (External and within langugage requirements)
• >=50% of functions written within a codebase should be documented according to its style (e.g. docstrings for R/python, help for MATLAB)

Testing

Some of these can be enforced by GitHub.

• If a package/compiled-code – successfully builds/installs.
• Some way of providing testing for the user
  • Provide/generate/point to data for them to use, with an example usage command
  • For new changes to be merged, all tests need to pass.

Explaination of requirements

• Pointers on how to add these things (TODO)
• Guiding principles (TODO)

Benefits of Bronze certification:

• Being able to cite the software in grant applications and publications! (DOI)
• Reasonable degree of certainty other will be able to install and use your code!

---

Silver Certification

BOLD text is different from Bronze

TODO: Insert example Silver cookie-cutter here.

Documentation

• Background/significance of program.
• Description of what program does is listed.
• A tutorial that showcases the program's usage. If multiple usages, examples of those are shown also.
• Description of all input parameters.
• Description of outputs.
  • Advanced usage
  • Document functions intended to be used by users.

Infrastructure

• Readme is present.
• DOI (Open Science Framework)
• License is present.
• Issues are enabled.
• Issue template(s) available (information requested by developers).

Code

• Installation instructions available.
• No excessive files included (gitignore).
• Adheres to a style guide (i.e., Python and Flake8)
• Portability guide
  • Application/Script: Containerization (Dockerfile and/or Singularity recipe)
  • Utility library: Up to user?

Testing

• Package successfully builds/installs.
• Some form of testing suite present.
  • Unit testing
  • Integration testing
  • Smoke tests
• Lines-hit tracker (>50%?)
• Provide testing for users

Benefits of Silver certification:

In addition to Bronze beneifts…

• Accessibility across a range of user experience.
• Greater portability of code (with containers) makes it more available to users, increasing the likelihood of your tool being used.
• Greater stability during development due to tests constantly checking code.

---

Gold Certification

BOLD text is different from Silver

TODO: Insert example Gold cookie-cutter here.

Documentation

• Background/significance of program.
• Description of what program does is listed.
• A tutorial that showcases the program's usage. If multiple usages, examples of those are shown also.
• Description of all input parameters.
• Description of outputs.
  • Advanced usage
  • Document functions intended to be used by users.
• Developer's guide
  • Docstrings / description of lower level API (functions, classes, etc)

Infrastructure

• Readme is present.
• [Digital Object Identifier] - Journal of Open Source Software submission
• License is present.
• Issues are enabled.
• Issue template(s) available (information requested by developers).
• Contribution Guide present.
• Code of Conduct present.

Code

• Installation instructions available.
• No excessive files included (gitignore).
• Adheres to a style guide (i.e., Python and Flake8)
• Portability guide
  • Application/Script: Containerization (Dockerfile and/or Singularity recipe)
  • Utility library: Up to user?

Testing

• Package successfully builds/installs.
• Some form of testing suite present.
  • Unit testing
  • Integration testing
  • Smoke tests
• Lines-hit tracker (>90%)
• Provide testing for users
• Benchmarking

Benefits of Gold certification:

In addition to Silver benefits…

• A greater sense of community guidelines, attracting contributors
• Extensive documentation assists users with more advanced use cases
• JoSS will increase the visibility of your project!

---

Examples

Docstrings

def read_json(json_file):
    """
    Validate and read a JSON file.
    
    Inputs
    ------
    json_file : file
    
    Outputs
    -------
    data : dict
        Data from JSON file
    """
    ...