Guidelines for our git and GitHub workflow

This document contains some guidelines on how we use git and GitHub at QuantStack.

Overview of git

TODO

Configuring git

TODO

Basic usage: contributing to an existing project

You will need to do the following every time you start working on an existing project, this needs to be done only once.

Fork the project

First, you need to create a fork of the original project, which we call upstream, under your Github account, we will call this fork origin.

For example, if you want to start contributing to Voila, go to https://github.com/voila-dashboards/voila and press the fork button then select your username.

Clone your fork

Now that you forked the project, you will need to clone it on your computer in order to start hacking. To do this, you will need to run the following command (replacing username by your Github name):

git clone https://github.com/username/voila.git

This will create a local voila directory containing your copy of the Voila repository.

cd voila

Setup git remote

Now you need to tell git what is the upstream repository.

git remote add upstream https://github.com/voila-dashboards/voila.git

If you run git remote -v, you should now see the upstream and origin urls:

origin	https://github.com/username/voila.git (fetch)
origin	https://github.com/username/voila.git (push)
upstream	https://github.com/voila-dashboards/voila.git (fetch)
upstream	https://github.com/voila-dashboards/voila.git (push)

Contribute to the project

Now that your local clone is setup, you can start hacking!

Important!
You should never commit changes to the master (or main) branch, unless you are making a release, but we'll come to it later.

Create a new branch from master

When adding a new feature or adding a bug-fix, the first thing to do is create a new branch:

git branch <branch_name>
git checkout <branch_name>

# Or the shortcut

git checkout -b <branch_name>

A set of good rules to follow for branches:

• It's good to have a meaningful branch name that explains what you are doing in it
• One branch = one bug fix or one feature
• Try to always create your branch from master, unless you have a good reason not to (when another branch is fixing an important bug that prevents you from testing your work)

Start developing

Now that your branch is created you can start hacking. I cannot help you in this section ;)

Visualize your current work

You can visualize your current work with:

git status

This will show you the files that are tracked by git and their status (modified/deleted), it will also show you the list of files that are here but not tracked by git:

On branch <branch_name>
Your branch is up to date with 'origin/<branch_name>'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	deleted:    package.json
	modified:   setup.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	docs/source/images/screenshots.zip
	foo.py

You can also see a diff with:

git diff
# or a diff of a specific file
git diff setup.py

Stage files before commiting

To stage a file for a commit, you can do

git add <filename>  # Include changes to filename for the next commit
git rm <filename>  # Remove filename in the next commit
git add <subdirectory> # Include all changes to files in subdirectory for the next commit

I recommend not using git add ., because this will add all untracked files which is usually not what you want.

Reverting a stage

If you added a file by mistake, you can remove it from the stage index with the following:

git reset filename

Commit your work

Now you can commit your work. In order to commit all staged files you can just do:

git commit

This will prompt for a commit message. It is important to give a meaningful commit message that explains what it does (e.g. "Fix bug" does not explain what it does, "Fix missing positional argument in sqrt function" is better).

You can also commit and fill the commit message with a single command:

git commit -m "<commit_message>"

Commit all tracked files (without staging manually)

In the case where you have too many files modified, it is very annoying to stage files one by one, instead you can ask git to commit all of your work:

git commit --all

This will commit all tracked files and discard untracked files.

Modify your latest commit

It is possible that you want to modify the message in your latest commit, or you want to add additional changes to it.

The --amend option is very useful in that case.

Say you forgot to add your changes to the setup.py file in your latest commit, you can just do:

git add setup.py
git commit --amend

This will prompt you for a commit message, you will see the old message and be able to modify it.

Listing the commits of your branch

git provides a command to list the commits of your branch:

git log

You can hit ENTER to scroll down. Use :q to exit.

Push your branch

Once you've commited all your changes, it is time to push them to your fork:

git push origin <branch_name>

Create a Pull Request (PR)

You can now go to the Github upstream repo (for instance https://github.com/voila-dashboards/voila), you should see a yellow rectangle allowing you to open a pull request, you can now click on "Compare & pull request".

If not, go the Github origin repo (i.e. your fork), select your branch in the dropdown menu on the left, then contribute and Open a Pull Request.

Explain what your Pull Request (PR) is about, you can reference issues or sometimes just follow the PR template provided by the repo. Then validate to open the PR.

Update your master branch

Once the maintainers of the project have merged the PR, your changes are available in the master (or main) branch of the project. It is time to update both your fork and your local clone:

git checkout master      # Back to the master branch
git pull upstream master # Updates your local clone with latest changes from upstream/master
git push origin master   # Push these changes to your fork

You can now safely delete the branch you used for hacking:

git branch -d <branch_name>

Advanced usage: Rebasing your branch

It may happen that your branch <branch_name> gets outdated and is behind master by a number of important commits (e.g. commits that fix the CI). In that case, maintainers may ask you to "rebase" your branch in order to update your PR before it can be merged.

Rebasing on master means applying all your commits on top of the current state of master. Rebasing also gives the opportunity to edit/reorder/squash commits.

Updating the master branch

In order to rebase, you will first need to update your master branch:

git checkout master
git pull upstream master
git push origin master

Rebasing

Then, check out your dev branch again, and rebase it:

git checkout <branch_name>
git rebase master

# Or the interactive version 
# (you will be prompted to edit/reorder/squash commits as wanted)

git rebase -i master

Git will try to apply each commit from <branch_name> to the master branch.

When a commit leads to a conflict that git cannot resolve by itself, the rebase process stops in an intermediate state, waiting for you to solve it. You will need to follow the next section on resolving conflicts, then resume the rebase:
Then solve the conflicts and save the file(s). When all the conflicts are solved, you can resume the rebase process:

git rebase --continue

Pushing the changes

After rebasing, you need to push the changes to your fork:

git push origin <branch_name> -f

Do not forget the -f option (for --force, otherwise git will output the following error message:

To prevent you from losing history, non-fast-forward updates were rejected

Aborting the rebase

If you want to abort the rebase process (for any reason), just run

git rebase --abort

This will restore the original state of <branch_name>

Bad practice

You may read some doc where people merge master into <branch_name> instead of rebasing <branch_name> onto master. We consider it as bad practice as it introduces meaningless merge commits and may make future rebase more complicated. Therefore git merge is highly discouraged at QuantStack.

Advanced usage: handling conflicts when opening a PR

Sometimes Github will complain it cannot open/merge a PR because your changes conflict with the files in the master branch. This is because some changes were merged into master after you created your branch and you started hacking. To solve this issue, you need to follow steps from the earlier section on "rebasing", then you will need to solve the conflicts locally when the rebase step prompts for solving conflicts.

Solving conflicts

To start the tool to solve conflict, run:

git mergetool

This will opened the merge tool you configured in .gitconfig.

Git will create a merge commit with your changes, and open a prompt so that you can enter a commit message.

NB: if you cannot solve the conflicts because you don't understand the changes made in master, then you should probably abort the rebase and ask for help to the author of the changes.

Advanced usage: removing/squashing commits from your branch

There are many reasons for removing commits from a branch: you pushed a lot of commits to debug and fix some CI errors, you want to gather commits together because they are related, etc.

First, checkout your branch:

git checkout <branch_name>

Then you will perform an interactive rebase:

git rebase -i master

Git will open a window listing all the commits of your branches that are not in the master branch. Then you can tell git what to do with each commit:

• pick: keep the commit
• drop: remove the commit
• squash: meld the commit into the previous one
• fixup: same as squash, but discard the commit message

Many other options are available and are described in the comment under the list of commits. When you are happy, save and exit. This will terminate the rebase.

NB: keep in mind than an interactive rebase is a rebase; therefore, if there are conflicts with the branch you rebase on, you will have to solve them to complete the rebase process.

NB: if you have a lot of commits in your branch and you know you only need to work on the last 5 commits, you can run the following

git rebase -i HEAD~5

instead of rebasing on master. Git will display only the last five commits in the prompt instead of all the commits in your branch since master.

Advanced usage: including changes from another branch

Picking a commit from another branch

Assume you created a branch for experimenting, let's call it exp_branch. Then you created another branch to develop a new feature, let's call it feat_branch. At some point, you realize that there is a commit in exp_branch that you need in feat_branch. You can pick the commit in feat_branch thanks to the cherry-pick feature.

First you need to get the hash of the commit.

git checkout exp_branch
git log # Note or copy the hash of the commit

Then you can pick the commit in your feat_branch:

git checkout feat_branch
git cherry-pick <commit_hash>
git log                       # you should see the new commit 

You don't need to provide the full hash of the commit, most of the times the first characters are enough.

Merging two branches into a single one

Assume you developed a feature in a dedicated branch, named base_feature. Then you started developing another feature in another branch, advanced_feature. At some point, you realise that advanced_feature needs everything form base_feature. Besides, opening a single PR with all the changes instead of one per branch would make more sense.

What you can do is rebase the advanced_feature branch on top of the base_feature branch. This way, the advanced_feature will contain the commits from the base_feature branch, followed by its own commits.

git checkout advanced_feature
git rebase base_feature
git log
# you should see commits from base_feature when scrolling down

If you pushed your branch before rebasing, you will need to force the next push:

git push origin advanced_feature -f

You can now safely delete the base_feature branch:

git branch -D base_feature

Notice the -D instead of -d. With -d, git would complain that the branch has not been merged into master and would prevent its deletion. -D forces git to delete it..

Advanced usage: working with another fork

Advanced usage: Stash changes

TODO

Advanced usage: Modifying your git config

You can modify your git config to improve your workflow. Your .gitconfig file should be located in your $HOME directory.

Editor

You can tell git which is your favorite editor for editing commit messages.

For example, if you want to use vim:

[core]
	editor = vim

Aliases

For slow-typists and lazy people, you can create your set of alias so you have less to type. It is up to you to decide for your aliases, but as an example I will give away my config:

[alias]
	co = checkout
	br = branch
	ci = commit -n
	st = status
	cl = clone
	sh = show
	up = "!git pull upstream master && git push origin master"
	last = log -4
	undo = reset HEAD~
	unstash = stash apply

This allows me to type git co master instead of git checkout master.

tags: trainings