GitLab course autumn 2021

Thursday Oct 21, ~~12:00~~ 12:35 - 15:00, right after the virtual NRIS meeting

Please register under Participants, click edit button top right

This document: https://hackmd.io/@hpc/gitlab-course
Last course material: https://gitlab.sigma2.no/training/git-course-for-metacenter
Survey results: https://docs.google.com/forms/d/1_P9yMW37a65rPgD83bl2-1ua6LT9md_RbwIlTyExKo8/edit#responses
Zoom: https://uit.zoom.us/j/69351724644

GitLab course autumn 2021
Suggestions for improvements

Participants

Please add yourself here if you plan to attend. If too few people register, we will need to postpone.

Geir
Radovan
Sabry
Espen FL
Jørgen
Stein Inge
Roger
Stig Rune (will drop in late)
Hicham
Dan
Vegard
Dhanya
pavel
Ole
Siri

Practical info

the audience this time is NRIS staff
each block is 30 min, but this including Q&A, discussion, so prepare to show for 15, max 20 minutes
zoom and this hackmd

For a future course or for a cheatsheet

GitLab Flavored Markdown

How to authenticate - Sabry

(20 min+ 20min try it your self)

Use case : I want to clone a repo from NRIS GitLab and modify it from my command line

Try to clone as usuall
token-based via https (Sabry)
ssh-keys (Sabry)

(D1) GitLab clone URL

(D2) How to clone

 /home/Git $ git clone git@gitlab.sigma2.no:training/git-temp-10-2021.git
Cloning into 'git-temp-10-2021'...
git@gitlab.sigma2.no's password: 
git@gitlab.sigma2.no: Permission denied (publickey,password).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

----
 /home/Git $ git clone https://gitlab.sigma2.no/training/git-temp-10-2021.git
Cloning into 'git-temp-10-2021'...
Username for 'https://gitlab.sigma2.no': gitlabuser1
Password for 'https://gitlabuser1@gitlab.sigma2.no': 
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.sigma2.no/training/git-temp-10-2021.git/'

(D3) Where is the SSH key?

#Assuming key already generated, if not run ssh-keygen first
 /home/Git $ cat $HOME/.ssh/id_rsa.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxp8x7lgwKeGZmlpvtrpIoXXXXXXXwvAbz00=user@jangama.uio.no

(D4) Gitlab SSH keys update

(D5) Clone with SSH keys

 /home/Git/temp $ git clone git@gitlab.sigma2.no:training/git-temp-10-2021.git
Cloning into 'git-temp-10-2021'...
remote: Enumerating objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 3
Receiving objects: 100% (3/3), done.
 /home/Git/temp $ ls
git-temp-10-2021

(D6) Create access token

Access tokens can be useful if you want to interact with the git repository via a script or cron job
For "normal work" as you yourself, ssh keys are often more convenient
Advantage of token is that they have a well defined life time. So if you are worried that the key may fall in wrong hands some time in future and you are working as root on something ultra-sensitive, token can be better.

(D7) Copy token

(D8) Use token from terminal

 /home/Git/temp $ TOKEN="u1uXX-";  git clone  https://git-temp-10-2021:$TOKEN@gitlab.sigma2.no/training/git-temp-10-2021.git
Cloning into 'git-temp-10-2021'...
remote: Enumerating objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 3
Unpacking objects: 100% (3/3), 224 bytes | 112.00 KiB/s, done.
 /home/Git/temp $ ls
git-temp-10-2021

Searching and finding (30 min) - Radovan

Suggestion for inspiration: https://coderefinery.github.io/git-intro/10-archaeology/

The web interface contains several search possibilities and search engines depending on where you are on the site:

Basic search
Issues
Merge requests (MR)
To-Do list
Projects
Groups
Issue boards
Code search
SHA search (Secure Hashing algorithm)

Overview of the projects (repos) and groups

Le Tour de Sigma2 groups and projects.

Finding a repository/project or issue or merge request

Go to https://gitlab.sigma2.no/
On top there is a search field
Try to find all projects/issues/MRs that talk about Saga and GPU
Search for a specific string, e.g. "we should fix"

Finding something within a repository

"I remember somewhere in the internal documentation there was something about local jump hosts - but where?"

Web interface:

visit https://gitlab.sigma2.no/sigma2/interndokumentasjon
on top search bar type "local jump host"

Command line:

git clone git@gitlab.sigma2.no:sigma2/interndokumentasjon.git
cd interndokumentasjon
git grep "local jump host"

Finding out when something changed

Our task:

On this page: https://documentation.sigma2.no/jobs/job_types/betzy_job_types.html
… with this source: https://gitlab.sigma2.no/sigma2/eksterndokumentasjon/-/blob/master/jobs/job_types/betzy_job_types.md
Text says that jobs are limited to 512 nodes
When was this changed? And who might know more about it?

Web interface:

https://gitlab.sigma2.no/sigma2/eksterndokumentasjon/-/blob/master/jobs/job_types/betzy_job_types.md
Click on "Blame" button (historical and very unfortunate naming)

Command line:

git clone git@gitlab.sigma2.no:sigma2/eksterndokumentasjon.git
cd eksterndokumentasjon
git annotate jobs/job_types/betzy_job_types.md

Discussion:

the when often more important than the who
the who can be useful to find out who might know more
not to blame anybody

Setting up and fixing a CI pipeline (30 min) - Espen

A bit of context

What is continuous integration (CI)

Offers simple of complex automation posibilities. Examples:
- Spell check content.
- Check and/or fix formatting of files.
- Check links and other dependencies.
- Build and test content (code, documentation, even whole infrastructures)
- Testing your and others efforts are key to CI, regardless of content. Try always to think about ways to test what you have.
Fully integrated into GitLab and very flexible.
There is also continuous deployment/delivery, which comes after the integration. We will not use time on defining what activites goes where and only use CI for the rest of this session.
Works rather similar on GitHub and Atlassian, which are alternatives to our GitLab.

GitLab and CI

Then utilizing CI one typically need a way to execute something or perform some operations.
- For documentation you need to for instance present it on a website. Sometimes this requires to execute some code, or at the very least move some files to the correct location.
- For code, you need to build it.
- Testing in general.
GitLab offers the concept of the Gitlab Runner, which in all essence is an isolated environment, which you can configure to run some operations.
The CI operations are typically called pipelines. One can have one or several pipelines for your content.

Let us try this in practice

What we will do

Address https://gitlab.sigma2.no/sigma2/eksterndokumentasjon/-/issues/203. Basically we need to add a spell checker to our external documentation: https://gitlab.sigma2.no/sigma2/eksterndokumentasjon.

Ironing out what we need

Documentation is mainly in markdown, so we need a spell checker that does not trigger on markdown syntax, only on the content. There are many spell checkers, so we are here using this:
- https://github.com/client9/misspell
When submitting updates to the documentation we want to get notified if there are any misspellings such that this is not accepted into the displayed documentation.
The user is responsible to fix the misspellings.
Documentation with spelling errors should not be published.

Let us do it

Create a branch where we can work with changes

Let us do it in the GUI to keep it less traditional. We know there is an issue, so let us call the branch issue/203.

Configure the CI in GitLab

In GitLab we configure CI in the .gitlab-ci.yaml file that is placed in your project root directory. The current .gitlab-ci.yaml for the external documentation is:

build: # Name of job
  image: python:3.7-alpine # Which container to run
  script:
  - pip install -r requirements.txt # Install prereqs in container
  - sphinx-build -W -b linkcheck . _build # Build documentation

What we need to do is that we will add a new job section which contains what we would like to do for the spellchecker. Consulting the manual of our chosen spell checker we see that we need to edit the .gitlab-ci.yaml file such that it is now (we ignore everything in this file which is relate to linkchecker):

stages:
  - spellchecker
  - build

spellchecker: # Name of job
  stage: spellchecker # Name of the stage the job is part of
  image: python:3.7-alpine # Which container to run
  script:
    - apk add --no-cache bash curl # Install curl etc.
    - curl -L -o ./install-misspell.sh https://git.io/misspell # Download
    - bash ./install-misspell.sh # Install
    - ./bin/misspell -error . # Run misspell on all files
    
build: # Name of job
  image: python:3.7-alpine # Which container to run
  script:
  - pip install -r requirements.txt # Install prereqs in container
  - sphinx-build -W -b linkcheck . _build # Build documentation

We should also fix another thing. It make sense not to even build the documentation if we locate a misspell, so we need to update the build job section above to:

build:
  stage: build
  when: on_success # Only run this job if all previous jobs is okey
  image: python:3.7-alpine
  script:
    - pip install -r requirements.txt
    - sphinx-build -W -b linkcheck . _build

The edit of .gitlab-ci.yaml is complete. Let us create a merge request (pull request on GitHub and Atlassian). In the merge request we make sure to give it a sensible title Adding spellchecker to the CI with the description Closes #204. Here we add the misspell spell checker as a job in the CI. We make the build job depend on its successful execution. and open the merge request.
Let us inspect the pipeline for the CI jobs. We see rather quickly that the spellchecker job fails and that the build job is skipped. Let us click on the spellchecker job and inspect the log. At the bottom we see that the spellchecker triggered on three misspellings:

jobs/interactive_jobs.md:101:12: "achived" is a misspelling of "achieved"
software/userinstallsw/python.md:160:5: "Supress" is a misspelling of "Suppress"
software/userinstallsw/python.md:160:13: "unneccessary" is a misspelling of "unnecessary"

Let us fix these misspellings and submit a new commit. Inspect pipeline again and observe that both the spellchecker and build completes and the merge request recieves a green mark which signifies that the tests, or checks have passed. Success.

Concluding remarks

As you can see this makes a more formal process and check of the spelling. If a user now submits something with spelling issues, we automatically get notified. Obviously one can expand this with many different jobs doing different things. But when approaching this, try to make it as simple as possible and build brick by brick. Also, for more information on valid parameters to set in the .gitlab-ci.yaml file, please consult https://docs.gitlab.com/ee/ci/yaml/.

Schedules

Sometimes you want a CI job to run at some specific time. Say week or something similar. Let us look at the linkchecker in the .gitlab-ci.yaml file:

linkchecker:
  stage: linkchecker
  image: python:3.7-alpine
  only:
    - schedules # This is the important part.
  script:
    - apk add --no-cache git bash curl
    - bash .ci/linkchecker.sh

Let us now go to CI/CD/Schedules in the left in the GUI. There we see that we have configured the Weekly run using a dedicated link checker. The linkchecker job will then run according to this schedule. For more information on this, please check: https://docs.gitlab.com/ee/ci/pipelines/schedules.html.

If you want to fully customize the time for the schedule you can use cron formatting. For details on how to format that, please check this https://en.wikipedia.org/wiki/Cron.

Discussions

Do you have some ideas and/or questions related to CI and how that could help you?

Project managenent (30 min) - Sabry

Issue - A task to be completed in a project
Milestone - A sprint. A time unit
Epic - A deliverable(s) may include one or many issues and involves one or many projects(Repos). Something that has a clear end.

Use case : I have done my part to solve an ongoing issue, now I need to transfer it to a colleague in another project to fix the rest

Use case : I to measure progress. How to manage capacity

Take home messages

Issues can move across projects and they should be closed at the end
Setting paramter like weight to issues makes it easy to plan progress and capacity
How to use visualization
Summarize best practices

Recovering from mistakes/situations (30 min) - Radovan

Merge conflict

Two branches modify same portion of the file in two different ways.

We can practice this:

https://gitlab.sigma2.no/training/merge-request-exercise
Create a new branch (in case we are missing permissions, we will adjust)
Exchange the line "Caramelized red onions" by something else
Create a merge request
We review the conflicts together

Committed to the wrong branch

Recovery from committing to master instead of side branch:

We made the commit on the server or already pushed the commit
- We can remove this commit but we probably shouldn't since other people and other branches and deployments may already depend on it
- Recovery on laptop: locally create a new commit with git revert somehash (somehash is the hash of the commit you want to revert)
- Recovery via GitLab: this can be done with the "Revert" button (click on commit, then "Options")
- git revert always creates a new commit, never modifies the past. It can also revert merge commits.
We made the commit only on the laptop, nobody has seen it yet
- Imagine the hash of the problem commit is "abcd123…"
- Create a branch myexperiment pointing to that commit. This is the branch you wish you had created first: git branch myexperiment abcd123
- Verify this: git log myexperiment: You should see that the new branch exists, that it contains the problem commit, and also that it is on par with master at this time.
- Now we can move master back. You can either move it back to a specific hash, but often what you want to do is to consider master read-only anyway and move it back to be on par with origin/master: git switch master; git reset --hard origin/master