# LIDA Data Scientist Enrichment Data Carpentry

This is a hackpad for the Data Carpentry workshop for the LIDA Data Scientist enrichment scheme 2022-2023. You can [edit this document](https://hackmd.io/@research-computing-leeds/lida-datasci-2022/edit?both) using [Markdown syntax](https://www.markdownguide.org/cheat-sheet/).

## Alex is striking Monday and Tuesday so will update this hackpad when he's back. Come say hello on the picket line by the Edge sports centre

## Overview

Over the course of the day we're going to try and accomplish the following:
- Hopefully, convince you reproducible research is important
- Highlight tools and best practices for:
- Organising your projects
- Managing project dependencies
- Touch on ideas for recording computational steps
- Look at how to share computational environments
- Convince you that version control is a tool you should use in your workflow
- To GitHub or not to GitHub
- The Research Data Management team in the library will talk to you about best practices for data management
- We'll cover some general sign posting of tools and resources for computational researchers at the University

## Agenda

| Time | Agenda | Materials
| --------- |----------------------------| ----- |
| 10:00 | Welcome, Sign posting, Reproducible Research Intro | [Organising Projects](https://coderefinery.github.io/reproducible-research/organizing-projects/), [Recording dependencies](https://coderefinery.github.io/reproducible-research/dependencies/)
| 11:00 | Break | |
| 11:15 | Managing dependencies (cont.) | [Recording dependencies](https://coderefinery.github.io/reproducible-research/dependencies/) |
| 12:00 | Lunch | |
| 13:00 | Library Data Team slot | [DMP template](https://bit.ly/UoLBasicDMP), [Sample DMP Robotics](https://bit.ly/RobotArmDMP), [Supporting materials in Minerva](https://minerva.leeds.ac.uk/webapps/blackboard/content/listContent.jsp?course_id=_423877_1&content_id=_7244662_1)|
| 14:00 | Break | |
| 14:15 | Intro to GitHub | [Remotes on GitHub](https://swcarpentry.github.io/git-novice/07-github/index.html) |
| 15:00 | Close |

## Who are we?

[Research Computing Team](https://arc.leeds.ac.uk/) and [Library Research Data Management Team](https://library.leeds.ac.uk/info/1600/about/149/meet_the_teams/2)

- Support computational researchers across campus
- Manage and support the [University HPC systems](https://arcdocs.leeds.ac.uk)
- Provide research software engineering consulting for research projects
- Support researchers access to national and regional computationally intensive resources
- Organise and engage the local computational researcher community
- Run [computational training](https://arc.leeds.ac.uk/training/courses/) for researchers

You can contact the research computing team directly via https://bit.ly/arc-help

## Resources for computational researchers

- [HPC services](https://arcdocs.leeds.ac.uk), high performance computing services available free at point of use to researchers. Includes some GPU resource. [Request an account](https://it.leeds.ac.uk/it?id=sc_cat_item&sys_id=4c002dd70f235f00a82247ece1050ebc)
- [LASER](https://lida-data-analytics-team.github.io/laserdocs/docs/laser_info/laser.html), secure data compute service provided by LIDA
- [Google Colab](https://colab.research.google.com/), google jupyter notebook service, possible to run GPU runtimes
- [Kaggle](http://kaggle.com/), online data science challenge website with notebook interfaces and GPU runtimes
- [Azure for students](https://azure.microsoft.com/en-us/free/students/), free £100 credit in Azure cloud for those with University account
- [GitHub](https://github.com/), remote repository for storing version-controlled code, also includes GitHub pages (free web hosting), GitHub actions (free compute runs)
- [The Carpentries](https://carpentries.org/), foundational coding and data science skills for researchers. Includes free lessons on R, Python and domain specific techniques.

## Tools

- [Conda Package Manager](https://docs.conda.io/en/latest/), tool for creating environments and managing package installation across a variety of languages
- [VSCode text editor](https://code.visualstudio.com/), an open source text editor from Microsoft, includes a large variety of plug-ins for linting, code checking and more
- [Git](https://git-scm.com/), a source control management tool. **Incredibly powerful tool**, helps track changes made to a project over time, you can roll back changes, develop alternate implementations. A way of saving different versions of your tool, integrates with source repositories like GitHub.
- [Lintr](https://cran.r-project.org/web/packages/lintr/readme/README.html), an R package for linting your R code, provides static analysis of code, highlights syntax errors, potential style errors.
- [Black](https://github.com/psf/black), Python code formatter, enforces the format of your python code inline with [PEP8](https://peps.python.org/pep-0008/)
- [Cookiecutter](https://github.com/cookiecutter/cookiecutter), a command line tool for creating project directory structures from templates

## Code snippets for the day

### Conda

Create a conda environement:

`conda create --name reproPython`

Activate the environment:

`conda activate reproPython`

Search for cookiecutter:

`conda search cookiecutter`

Install cookiecutter:

`conda install -c conda-forge cookiecutter`

### Cookiecutter

Log of stuff we've done with Cookiecutter:

# Change directory to your desktop for this test
$ cd %USERPROFILE%/Desktop
# Install git
$ conda install -y git
# Create a template project with Cookiecutter
# Fill in all the suggested details (we went with repro-python as the name)
$ cookiecutter gh:mkrapp/cookiecutter-reproducible-science
# Go into our new project
$ cd repro-python
# Initialise our git repo
$ git init
# Set the primary branch name
$ git branch -m main
# Look at the current state of the repository
$ git status
# Add everything to our repo as an initial commit
$ git add .
# Commit these changes
$ git commit -m "Initial Commit"

### VSCode and editing files

In the afternoon we looked at the git cycle in more detail and installed a text editor tool to help us edit files.

There are many [text editors](https://en.wikipedia.org/wiki/List_of_text_editors) out there (with many [opinionated perspectives](https://en.wikipedia.org/wiki/Editor_war) on which is the best). We're going to use [VSCode](https://code.visualstudio.com/) an open source, pluggable text editor from Microsoft. You can again install this directly from the website without admin rights, so go ahead and install it.

You can watch a good intro guide to VSCode through their [intro video](https://code.visualstudio.com/docs/introvideos/basics).

Steps we took:

1. Open our project within VSCode
You can do this in a number of ways.
1. From the terminal you can `cd` into the directory you want to open and run `code .` (you'll need to close and reopen your terminal after installing VSCode for this to work). This should open the VSCode session in your current directory.
2. Using the menu in VSCode click `File > Open Folder` and browse through the file explorer to the folder/directory you want to open

2. We made an edit to a file
We didn't do anything fancy, just made a change to a file.
Because we're already run `git init` in this directory any changes we make to files we've previously committed will be shown when we run `git status`

3. Start the git workflow
To get started with the git workflow we need to swap back to our Terminal. We can run `git status` to check if git detects any changes, if it does we can add these changes by running `git add FILENAME` where `FILENAME` is the name of the file that's changed. This stages the files ready to be committed (snapshotted).

4. Commit your changes
Once the changes have been added (and we like to commit things in chunks rather than squashing together lots of changes into a single commit) we run `git commit -m "some message"` where you leave an informative message about the changes you've made.

This is the basic cycle of the git workflow and forms the basis of an approach to writing your code that captures changes you make gradually helping with reproducibility. You can read more about using git with the [Git Novice tutorial](https://swcarpentry.github.io/git-novice/).

#### Integrating git and GitHub

Finally, we looked at how to integrate git, the tool for storing changes to our code locally, and GitHub, an online platform for hosting our code and it's associated history.

Integrating GitHub into our workflow is another super useful tool for productivity and reproducibility. It allows you to upload the current history of the code you've been working on to an online platform so that if you lose your computer or move onto another machine you can download (clone) the code and all associated history onto your new device. It's also the no. 1 site for sharing code and makes it easy to open source your code and it's development history. You can also keep your code private and control who has access to view or edit your repositories.

Here's a summary of the steps we took to create a GitHub repository for our reproducible python project and upload (push) our local code and it's associated history of changes up to this repository.

1. To get started we needed to have GitHub accounts, which you can sign up for for free using your University email address.

2. Install the GitHub command line interface (CLI), we can do this through conda with the command:

    $ conda install -c conda-forge gh

3. Authenticate our GitHub account to use locally with git.
For GitHub to know who we are we need to authenticate ourselves with GitHub. This means confirming which user account we are and associating those details with our local git client. Git and GitHub are two separate tools so we have to take steps to associate each with the other in this way.

    $ gh auth login

    ? What account do you want to log into? GitHub.com
    ? What is your preferred protocol for Git operations? HTTPS
    ? Authenticate Git with your GitHub credentials? Yes
    ? How would you like to authenticate GitHub CLI? Login with a web browser

    ! First copy your one-time code: 6E09-F247
    Press Enter to open github.com in your browser...
    ✓ Authentication complete.
    - gh config set -h github.com git_protocol https
    ✓ Configured git protocol
    ✓ Logged in as Sparrow0hawk

Here you can see running `gh auth login` takes us through a step by step process prompting us for input at various stages. We select to authenticate through a web browser and are prompted to login using our GitHub account details and provide the 8 letter code provided.

At some stage a pop up credential manager may appear
At this stage select `store` and if re prompted select `store` again.

Once this process completes you have configured your local git client to authenticate with your GitHub account 🎉

4. Pushing local code to GitHub
So now we've got a git configured local project folder, and configured our local git client to associate with our GitHub account. Now we need to get our local code onto GitHub!

The `gh` command line tool makes this straight forward although we can create repositories on [GitHub directly in the web interface](https://docs.github.com/en/get-started/quickstart/create-a-repo).

We use the `gh repo create` command to create a new GitHub repository, this will serve as the copy on GitHub of all our local project code and its history. Again it walks us step by step through the process of creating this repository

    $ gh repo create 

    ? What would you like to do?  [Use arrows to move, type to filter]
    > Create a new repository on GitHub from scratch
      Push an existing local repository to GitHub

Here we select `Push an existing local repository to GitHub` because we've already started working with our local repository and just want to copy it up to GitHub.

    ? Path to local repository (.)

Here it asks for the path to the local repository, it defaults to our current working directory which is what we want as we've `cd` into our project folder.

    ? Repository name (repro-py)
    ? Description (repro-py)

Next it asks for a repository name (defaulting to the directory name we've specified above) and a description (which we leave blank for now).
Next it asks about visibility of the repository, i.e. how accessible do we want this code to be on GitHub.

    ? Visibility  [Use arrows to move, type to filter]
    > Public
      Private
      Internal

It defaults to public which means anyone can view our code, you can change this as desired.
We get a confirmation that we have now created an empty repository on GitHub using the settings provided and now follow the steps for configuring our local repository to connect to GitHub.

    ✓ Created repository Sparrow0hawk/repro-py on GitHub
    ? Add a remote? (Y/n)

Adding a remote is a way of connecting our local repository to GitHub, it essentially specifies the location of the remote repository on GitHub so that when we _push_ (push being the git term for updating any local changes we've made with our GitHub copy) code it sends it up to GitHub.
Next it asks us to specify a name for the remote, the default being `origin` which we will leave it as.

    ? What should the new remote be called? (origin)

After it configures the remote it now asks if we want to push our existing history up to GitHub.

    ✓ Added remote https://github.com/Sparrow0hawk/repro-py.git
    ? Would you like to push commits from the current branch to "origin"? (Y/n)

If we select `Y` here it will copy all of the existing code and associated history up to our GitHub repository. This is useful when starting out to sync everything but the steps we've run through here are one-off setup steps. To practice the more regular commands select `n` here.

5. Keeping our local and GitHub repositories in sync

The more common way we push changes from our local repository to GitHub is with the command `git push`. This pushes (uploads) any commits we've got locally that aren't present on GitHub.

We can also download any commits on GitHub that aren't present locally with the command `git pull`.

## Questions and Answers

Please do put any questions you like in here, or shout them out in person.

### If I install Anaconda with admin rights, I can't do everything I want. Is there a better way?

If you install anaconda using admin rights on Windows, or install it using apt/dnf/yum on Linux, you end up with a setup that's actually not quite as easy to use as if you install it as a normal user.

Download the installer from the miniconda/anaconda website, and install it as a normal user into you home directory, and that's actually the best install for a normal user.