# GitLab course autumn 2021

:::danger
Thursday Oct 21, ~~12:00~~ 12:35 - 15:00, right after the virtual NRIS meeting

Please register under [Participants](###Participants), click edit button top right
:::

:::info
- This document: https://hackmd.io/@hpc/gitlab-course
- Last course material: https://gitlab.sigma2.no/training/git-course-for-metacenter
- Survey results: https://docs.google.com/forms/d/1_P9yMW37a65rPgD83bl2-1ua6LT9md_RbwIlTyExKo8/edit#responses
- Zoom: https://uit.zoom.us/j/69351724644
:::

---
[TOC]

---
### Participants

Please add yourself here if you plan to attend. If too few people register, we will need to postpone.

- Geir
- Radovan
- Sabry
- Espen FL
- Jørgen
- Stein Inge
- Roger
- Stig Rune (will drop in late)
- Hicham
- Dan
- Vegard
- Dhanya
- pavel
- Ole
- Siri

### Practical info

- the audience this time is NRIS staff
- each block is 30 min, but this including Q&A, discussion, so prepare to show for 15, max 20 minutes
- zoom and this hackmd

### For a future course or for a cheatsheet

- [GitLab Flavored Markdown ](https://docs.gitlab.com/ee/user/markdown.html)

### How to authenticate - Sabry
- (20 min+ 20min try it your self)
:::success
**Use case** : I want to clone a repo from NRIS GitLab and modify it from my command line
:::

- Try to clone as usuall
- token-based via https (Sabry)
- ssh-keys (Sabry)

graph LR
 
 subgraph "Terminal3"
 SubGraph3 -- HTTPS -->  Node13["TOKEN='XX';git clone  https://git-temp-10-2021:$[e-mail]/training/git-temp-10-2021.git"]
 Node13--> Node14("Success (D8)")
 end
  
 subgraph "Terminal2"
 SubGraph2 -- SSH -->  Node9["git clone [e-mail]:training/git-temp-10-2021.git"]
 Node9--> Node10("Success (D5)")
 SubGraph2 -- HTTPS -->  Node11["git clone https://gitlab.sigma2.no/training/git-temp-10-2021.git"]
 Node11--> Node12("Authentication error")
 end

    
subgraph "Terminal"
SubGraph1 -- SSH -->  Node2["git clone [e-mail]:training/git-temp-10-2021.git (D1)"]
SubGraph1  -- HTTPS --> Node4["git clone https://gitlab.sigma2.no/training/git-temp-10-2021.git (D3)"]
Node2 --> Node3["Authentication error (D2)"]
end

  subgraph "Gitlab web"
  Node1[NRIS GitLab,D1] -->SubGraph1[From terminal No keys]
  Node4 --> Node3
  Node1 --> Node5["Set up ssh-keys  (D3, D4)"]
  Node5 --> SubGraph2[From terminal SSH keys]
  Node1 --> Node6["Set up access tokens  (D6, D7)"]
  Node6 --> SubGraph3[From terminal acess token]
  end

#### (D1) GitLab clone URL
<img src="https://i.imgur.com/wCZtsSm.png" alt="drawing" width="400"/>

#### (D2) How to clone

 /home/Git $ git clone [e-mail]:training/git-temp-10-2021.git
Cloning into 'git-temp-10-2021'...
[e-mail]'s password: 
[e-mail]: Permission denied (publickey,password).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

----
 /home/Git $ git clone https://gitlab.sigma2.no/training/git-temp-10-2021.git
Cloning into 'git-temp-10-2021'...
Username for 'https://gitlab.sigma2.no': gitlabuser1
Password for 'https://[e-mail]': 
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.sigma2.no/training/git-temp-10-2021.git/'

#### (D3) Where is the SSH key?

#Assuming key already generated, if not run ssh-keygen first
 /home/Git $ cat $HOME/.ssh/id_rsa.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxp8x7lgwKeGZmlpvtrpIoXXXXXXXwvAbz00=[e-mail]

#### (D4) Gitlab SSH keys update
<img src="https://i.imgur.com/kFv9C3R.png" alt="create key" width="400"/>

#### (D5) Clone with SSH keys

 /home/Git/temp $ git clone [e-mail]:training/git-temp-10-2021.git
Cloning into 'git-temp-10-2021'...
remote: Enumerating objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 3
Receiving objects: 100% (3/3), done.
 /home/Git/temp $ ls
git-temp-10-2021

#### (D6) Create access token

<img src="https://i.imgur.com/gpabOc3.png" alt="Create token" width="400"/>

- Access tokens can be useful if you want to interact with the git repository via a script or cron job
- For "normal work" as you yourself, ssh keys are often more convenient
- Advantage of token is that they have a well defined life time. So if you are worried that the key may fall in wrong hands some time in future and you are working as root on something ultra-sensitive, token can be better.

#### (D7) Copy token
<img src="https://i.imgur.com/BRiUeov.png" alt="Copy token" width="400"/>

#### (D8) Use token from terminal

 /home/Git/temp $ TOKEN="u1uXX-";  git clone  https://git-temp-10-2021:$[e-mail]/training/git-temp-10-2021.git
Cloning into 'git-temp-10-2021'...
remote: Enumerating objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 3
Unpacking objects: 100% (3/3), 224 bytes | 112.00 KiB/s, done.
 /home/Git/temp $ ls
git-temp-10-2021

### Searching and finding (30 min) - Radovan

Suggestion for inspiration: https://coderefinery.github.io/git-intro/10-archaeology/

The web interface contains several search possibilities and search engines depending on where you are on the site:
- Basic search
- Issues
- Merge requests (MR)
- To-Do list
- Projects
- Groups
- Issue boards
- Code search
- SHA search (Secure Hashing algorithm)

#### Overview of the projects (repos) and groups

Le Tour de [Sigma2](https://gitlab.sigma2.no/) groups and projects.

#### Finding a repository/project or issue or merge request

- Go to https://gitlab.sigma2.no/
- On top there is a search field
- Try to find all projects/issues/MRs that talk about Saga and GPU
- Search for a specific string, e.g. "we should fix"

#### Finding something within a repository

"I remember somewhere in the internal documentation there was something about local jump hosts - but where?"

Web interface:
- visit https://gitlab.sigma2.no/sigma2/interndokumentasjon
- on top search bar type "local jump host"

Command line:
- `git clone [e-mail]:sigma2/interndokumentasjon.git`
- `cd interndokumentasjon`
- `git grep "local jump host"`

#### Finding out when something changed

Our task:
- On this page: https://documentation.sigma2.no/jobs/job_types/betzy_job_types.html
- ... with this source: https://gitlab.sigma2.no/sigma2/eksterndokumentasjon/-/blob/master/jobs/job_types/betzy_job_types.md
- Text says that jobs are limited to 512 nodes
- **When was this changed?** And who might know more about it?

Web interface:
- https://gitlab.sigma2.no/sigma2/eksterndokumentasjon/-/blob/master/jobs/job_types/betzy_job_types.md
- Click on "Blame" button (historical and very unfortunate naming)

Command line:
- `git clone [e-mail]:sigma2/eksterndokumentasjon.git`
- `cd eksterndokumentasjon`
- `git annotate jobs/job_types/betzy_job_types.md`

Discussion:
- the *when* often more important than the *who*
- the *who* can be useful to find out who might know more
- not to blame anybody

### Setting up and fixing a CI pipeline (30 min) - Espen

#### A bit of context

##### What is continuous integration (CI)

- Offers simple of complex automation posibilities. Examples:
- Spell check content.
- Check and/or fix formatting of files.
- Check links and other dependencies.
- Build and test content (code, documentation, even whole infrastructures)
- Testing your and others efforts are key to CI, regardless of content. Try always to think about ways to test what you have.
- Fully integrated into GitLab and very flexible.
- There is also continuous deployment/delivery, which comes after the integration. We will not use time on defining what activites goes where and only use CI for the rest of this session.
- Works rather similar on GitHub and Atlassian, which are alternatives to our GitLab.

##### GitLab and CI

- Then utilizing CI one typically need a way to execute something or perform some operations.
- For documentation you need to for instance present it on a website. Sometimes this requires to execute some code, or at the very least move some files to the correct location.
- For code, you need to build it.
- Testing in general.
- GitLab offers the concept of the Gitlab Runner, which in all essence is an isolated environment, which you can configure to run some operations.
- The CI operations are typically called pipelines. One can have one or several pipelines for your content.
-
#### Let us try this in practice

##### What we will do

Address https://gitlab.sigma2.no/sigma2/eksterndokumentasjon/-/issues/203. Basically we need to add a spell checker to our external documentation: https://gitlab.sigma2.no/sigma2/eksterndokumentasjon.

##### Ironing out what we need

- Documentation is mainly in markdown, so we need a spell checker that does not trigger on markdown syntax, only on the content. There are many spell checkers, so we are here using this:
- https://github.com/client9/misspell
- When submitting updates to the documentation we want to get notified if there are any misspellings such that this is not accepted into the displayed documentation.
- The user is responsible to fix the misspellings.
- Documentation with spelling errors should not be published.

##### Let us do it

###### Create a branch where we can work with changes

Let us do it in the GUI to keep it less traditional. We know there is an issue, so let us call the branch `issue/203`.

###### Configure the CI in GitLab

1. In GitLab we configure CI in the `.gitlab-ci.yaml` file that is placed in your project root directory. The current `.gitlab-ci.yaml` for the external documentation is:

build: # Name of job
  image: python:3.7-alpine # Which container to run
  script:
  - pip install -r requirements.txt # Install prereqs in container
  - sphinx-build -W -b linkcheck . _build # Build documentation

2. What we need to do is that we will add a new job section which contains what we would like to do for the spellchecker. Consulting the manual of our chosen spell checker we see that we need to edit the `.gitlab-ci.yaml` file such that it is now (we ignore everything in this file which is relate to `linkchecker`):

stages:
  - spellchecker
  - build

spellchecker: # Name of job
  stage: spellchecker # Name of the stage the job is part of
  image: python:3.7-alpine # Which container to run
  script:
    - apk add --no-cache bash curl # Install curl etc.
    - curl -L -o ./install-misspell.sh https://git.io/misspell # Download
    - bash ./install-misspell.sh # Install
    - ./bin/misspell -error . # Run misspell on all files
    
build: # Name of job
  image: python:3.7-alpine # Which container to run
  script:
  - pip install -r requirements.txt # Install prereqs in container
  - sphinx-build -W -b linkcheck . _build # Build documentation

3. We should also fix another thing. It make sense not to even build the documentation if we locate a misspell, so we need to update the `build` job section above to:

build:
  stage: build
  when: on_success # Only run this job if all previous jobs is okey
  image: python:3.7-alpine
  script:
    - pip install -r requirements.txt
    - sphinx-build -W -b linkcheck . _build

4. The edit of `.gitlab-ci.yaml` is complete. Let us create a merge request (pull request on GitHub and Atlassian). In the merge request we make sure to give it a sensible title `Adding spellchecker to the CI` with the description `Closes #204. Here we add the misspell spell checker as a job in the CI. We make the build job depend on its successful execution.` and open the merge request.

5. Let us inspect the pipeline for the CI jobs. We see rather quickly that the `spellchecker` job fails and that the `build` job is skipped. Let us click on the `spellchecker` job and inspect the log. At the bottom we see that the spellchecker triggered on three misspellings:

jobs/interactive_jobs.md:101:12: "achived" is a misspelling of "achieved"
software/userinstallsw/python.md:160:5: "Supress" is a misspelling of "Suppress"
software/userinstallsw/python.md:160:13: "unneccessary" is a misspelling of "unnecessary"

6. Let us fix these misspellings and submit a new commit. Inspect pipeline again and observe that both the `spellchecker` and `build` completes and the merge request recieves a green mark which signifies that the tests, or checks have passed. Success.

##### Concluding remarks

As you can see this makes a more formal process and check of the spelling. If a user now submits something with spelling issues, we automatically get notified. Obviously one can expand this with many different jobs doing different things. But when approaching this, try to make it as simple as possible and build brick by brick. Also, for more information on valid parameters to set in the `.gitlab-ci.yaml` file, please consult https://docs.gitlab.com/ee/ci/yaml/.

#### Schedules

Sometimes you want a CI job to run at some specific time. Say week or something similar. Let us look at the `linkchecker` in the `.gitlab-ci.yaml` file:

linkchecker:
  stage: linkchecker
  image: python:3.7-alpine
  only:
    - schedules # This is the important part.
  script:
    - apk add --no-cache git bash curl
    - bash .ci/linkchecker.sh

Let us now go to CI/CD/Schedules in the left in the GUI. There we see that we have configured the `Weekly run using a dedicated link checker`. The `linkchecker` job will then run according to this schedule. For more information on this, please check: https://docs.gitlab.com/ee/ci/pipelines/schedules.html.

If you want to fully customize the time for the schedule you can use `cron` formatting. For details on how to format that, please check this https://en.wikipedia.org/wiki/Cron.

#### Discussions

Do you have some ideas and/or questions related to CI and how that could help you?

### Project managenent (30 min) - Sabry

:::info
**Issue** - A task to be completed in a project
**Milestone** - A sprint. A time unit
**Epic** - A deliverable(s) may include one or many issues and involves one or many projects(Repos). Something that has a clear end.
:::

:::success
**Use case** : I have done my part to solve an ongoing issue, now I need to transfer it to a colleague in another project to fix the rest
:::
:::success
**Use case** : I to measure progress. How to manage capacity
:::

#### Take home messages

* Issues can move across projects and they should be closed at the end
* Setting paramter like weight to issues makes it easy to plan progress and capacity
* How to use visualization
* Summarize best practices

### Recovering from mistakes/situations (30 min) - Radovan

#### Merge conflict

Two branches modify same portion of the file in two different ways.

We can practice this:
- https://gitlab.sigma2.no/training/merge-request-exercise
- Create a new branch (in case we are missing permissions, we will adjust)
- Exchange the line "Caramelized red onions" by something else
- Create a merge request
- We review the conflicts together

#### Committed to the wrong branch

Recovery from committing to `master` instead of side branch:
- We made the commit on the server or already pushed the commit
- We can remove this commit but we probably shouldn't since other people and other branches and deployments may already depend on it
- Recovery on laptop: locally create a **new commit** with `git revert somehash` (somehash is the hash of the commit you want to revert)
- Recovery via GitLab: this can be done with the "Revert" button (click on commit, then "Options")
- `git revert` always creates a new commit, never modifies the past. It can also revert merge commits.
- We made the commit only on the laptop, nobody has seen it yet
- Imagine the hash of the problem commit is "abcd123..."
- Create a branch `myexperiment` pointing to that commit. This is the branch you wish you had created first: `git branch myexperiment abcd123`
- Verify this: `git log myexperiment`: You should see that the new branch exists, that it contains the problem commit, and also that it is on par with `master` at this time.
- Now we can move `master` back. You can either move it back to a specific hash, but often what you want to do is to consider `master` read-only anyway and move it back to be on par with `origin/master`: `git switch master; git reset --hard origin/master`

Discussion:
- Motivation why rollback functionality is useful. Example: cluster DevOps using continuous integration and continuous deployment
- One possible solution:
- Changes are merge request towards `staging` branch
- By merging `staging` into `master` or `main` we can put things into production
- We can roll back since every change is a commit
- We can find out when a change was introduced and why (commit message)
- What benefits do you see when working on a branch and changing via review compared to working on `master`?

# Suggestions for improvements