---
title: 'Unifying Konveyor'
---

Unifying Konveyor project code conventions
===

[TOC]

## Problem statement
Within number of Konveyor projects and their repositories both [upstream](https://github.com/orgs/konveyor/repositories) and [downstream](https://gitlab.cee.redhat.com/mig-integration) there are no common code conventions and development workflows that would ease the contribution process.

Almost every project requires duplicated effort in automating very simple tasks such as creating development environment, building the source code, running simple test both locally and in the CI ecosystem, generating documentation, etc.

Even minor things like documentation have different naming / paths and it's inconsistent e.g. development instrutions: [docs/Development.md](https://github.com/konveyor/pelorus/blob/master/docs/Development.md) vs [docs/hacking.md](https://github.com/konveyor/mig-operator/blob/master/docs/hacking.md) vs [README.md](https://github.com/konveyor/forklift-ui#development-scripts) vs [contributing.md](https://github.com/konveyor/move2kube/blob/main/contributing.md) and [code-of-conduct.md](https://github.com/konveyor/move2kube/blob/main/code-of-conduct.md)

This leads to more work, maintenance and differences for the parts that could be similar.

Also aspect of projects belonging to one community is reduced by those inconsistencies.

## Proposal
Create a set of common conventions that could be used in automated fashion. For the parts that are not automated the docs should be available for developers to follow.

# Execution
There are couple of areas to address. Proposals are described below. Most of the scripting and development should be stored in one common repository that is available from both upstream and downstream, meaning that [GitHub](https://github.com/konveyor/) is the first choice to consider.

graph
    GitHub_project --> common_repository;
    GitLab_project --> common_repository;

## Common way of creating dev env

Every developer who would like to contribute to the Konveyor project should be able to prepare local developer environment with all needed dev and test deps in similar way.

Proposal is to have a common syntax invoked from the `Makefile`

# To create development environment
make dev-env

# To tear down developmen environment
make clean-dev-env

We should include snippet for the Makefile together with supporting scripts that allows to install packages from different locations and packaging systems. e.g. `PyPI requirements.txt, test-requirements.txt, Go, dnf, GitHub released binaries, OpenShift CLIs, Ansible roles, collections, etc...`

## Common way of preparing test environment

Simple static code analysis as well as simple test runs that could be ran on the local developer box as well as part of the CI/CD systems should have common way kind of ***entry point*** of running those. This allows to detach test logic from any system runner.

It could be the same as previously mentioned `make dev-env`, but also something more specific to the test scenario.

The repository that holds such test environments should be language agnostic and support different use cases that are adopted in the Konveyor set of projects.

## Common way of running various tests

Each code analysis test as well test frameworks that could be shared *`NOTE that not all of them can be shared and those should live in the per project repository`* should be ran in a similar way.

Proposal is similar to the dev-env and have common section/target invoked from the `Makefile`. One special target ***test*** should run all the tests that are required prior to submission code as the PR, example:

.PHONY tests
tests: \
    # code to run all tests via pre-commit.
    # This may be just a list of Makefile targets 
    # 'pylava lint shellcheck' however running them
    # via pre-commit will allow to catch all the errors
    # at once rather then failing entire run on a first error.

.PHONY: pylava lint shellcheck
pylava:
    # code to run pylava via pre-commit

lint:
    # code to run lint via pre-commit

shellcheck:
    # code to run shellcheck via pre-commit

Having such entry point will allow us to invoke tests locally and inside CI/CD systems in a very same way, hiding logic behind the scenes by simply running:

# To run all tests
make tests

# Or if individual test run is required
make shellcheck

## Common configuration files for code analysis

To ensure code is consistent across different Konveyor projects we should store configuration files for each of the linters/tests that will then be used in various projects. Those configuration files should be for various programming languages and selected analysis tools. Of course if the file contains information about project folder structure or files then only applicable rules should be common e.g.:

.ansible-lint
pylava.ini
pyproject.toml
lintconf.yaml
[...]

## Carefully selected set of pre-commit hooks

[pre-commit](https://pre-commit.com/) is a framework/package manager that allows to easilly install and run multi-language set of static code analysis tools agains the code. This framework can be also used to run a bit more comprehensive set of tests.

There is a wide community supported set of [hooks](https://pre-commit.com/hooks.html) that we should agree on and enable for the applicable konveyor projects storing the configuration files in the common repository.

If there are some missing code test runners or available one is not aligned with our requirements (`for example not accepted hook as it's modifying the source code`) we will write our own version and store in the common repository.

We will store .pre-commit-config.yaml that lists *all* of the selected hooks for all the projects, with manual stage set. This stage will be commented out if the project needs it, e.g.:

- repo: https://github.com/golangci/golangci-lint
  rev: v1.44.2
  hooks:
    - id: golangci-lint
#      stages: [manual]

Those hooks will be then wrapped around Makefile to allow common entry point accross different projects.

# Example scenarios

This section lists example scenarios and it's meant to explain better the entire concept.

### Running the tests via Makefile

flowchart LR
CI["CI system (github actions/Praw/gitlab runner)"] --> |make tests|id1
Developer --> |make tests|id1
id1 --> |pre-commit run|pre-commit
subgraph "Konveyor Project"
id1{{Makefile}}
end
subgraph "pre-commit"
id2{{pylint}}
id3{{shellcheck}}
id4{{other hook runner}}
end

### Running the tests prior to git push
Action done via pre-commit hook installed by the ```pre-commit``` CLI

flowchart LR
subgraph Konveyor_project
direction LR
git-commit{{git commit}} --> id1
id1{{pre-commit github hook}}
end
subgraph pre-commit
id2{{pylint}}
id3{{shellcheck}}
id4{{other hook runner}}
end
Konveyor_project --> |pre-commit run|pre-commit

### Running individual test

flowchart LR
      CI["CI system (github actions/Praw/gitlab runner)"] --> |make shellcheck|id1
      Developer --> |make shellcheck|id1
      id1 -->  |pre-commit run shellcheck|pre-commit
      subgraph "Konveyor Project"
        id1{{Makefile}}
      end    
      subgraph "pre-commit"
        id2{{shellcheck}}
      end

### Code analysis config files
Most files are stored in the common repository and synced. Possibly using some `Makefile sync-configs target`. Konveyor Project may even not contain those files and they will be synced before actual run in the test preparation step. Konveyor Project may also store a modified version of config file which is templated from common repository.

flowchart LR
    subgraph Common_repository
      direction LR
      id1(".ansible-lint\npylava.ini\npyproject.toml\nlintconf.yaml\n[...]")
    end
    subgraph Konveyor_Project
      id2{{Makefile}}
      id3{{.ansible-lint\nlintconf.yaml}}
end
    id1 -.-> |sync|id3

### Makefile and scripts
`Makefile` and scripts which may live in the common repository should be synced with the Konveyor Project.
All pre-check targets, even those that are not relevant for the programming language should be listed in the Makefile for easy sync.

***Note*** *the common_repository `Makefile` is a subset of Konveyor project*

flowchart LR
    subgraph Common_repository
      direction LR
      id1("Makefile")
      id2("scipts/")
    end
    subgraph Konveyor_Project
      id3{{Makefile}}
      id4{{.ansible-lint\nlintconf.yaml}}
      id5{{scripts/}}
end
    id1 -.-> |"sync (subset)"|id3
    id2 -.-> |"sync"|id5

### Custom pre-commit hooks
Similarly to any hook a custom hook will be written and stored in the common repository. Writing a custom hook is relatively easy task and a number of programming languages are supported. Please refer to the [creating new hooks ](https://pre-commit.com/index.html#new-hooks).

Running a custom hook is same as other ones

flowchart LR
CI["CI system (github actions/Praw/gitlab runner)"] --> |make shellcheck|id1
Developer --> |make customhook|id1
id1 --> |pre-commit run customhook|pre-commit
subgraph "Konveyor Project"
id1{{Makefile}}
end
subgraph "pre-commit"
id2{{shellcheck}}
end

### Putting everything together

flowchart LR
subgraph ida ["Konveyor GitHub projects"]
direction LR
subgraph id1a ["Pelorus"]
direction LR
id1b>"python, helm, shell, docs"]
id1c{{"Makefile"}}
end
subgraph id2a ["Crane Operator"]
direction LR
id2b>"ansible, podman"]
id2c{{"Makefile"}}
end
subgraph id3a ["Move2Kube"]
direction LR
id3b>"go, shell, podman"]
id3c{{"Makefile"}}
end
subgraph id4a ["Other..."]
direction LR
id4b>"Fortran"]
id4c{{"Makefile"}}
end
end

subgraph idb ["Konveyor GitLab projects"]
direction LR
subgraph id5a ["minikube-cluster"]
direction LR
id5b>"ansible"]
id5c{{"Makefile"}}
end
subgraph id6a ["oadp-operator-test"]
direction LR
id6b>"ansible, python"]
id6c{{"Makefile"}}
end
subgraph id7a ["Other..."]
direction LR
id7b>"go"]
id7c{{"Makefile"}}
end
end

subgraph idc ["Common Repository"]
id9a{{"Makefile"}}
subgraph id9x ["Config files"]
id9b(".ansible-lint\npylava.ini\npyproject.toml\nlintconf.yaml\ngolint.yaml\n[...]")
end
subgraph id9h ["Custom pre-check"]
direction LR
id9g>"fortran-check"]
end
subgraph id7a ["Other..."]
direction LR
id7b>"go"]
id7c{{"Makefile"}}
end
end
subgraph id10a ["pre-commit.com supported hooks"]
direction LR
id10b{{"ansible lint\ncheck yaml\ngolint\n[...]"}}
end

id9a -.-> |"sync"|ida
id9a -.-> |"sync"|idb
id9x -.-> |"sync"|ida
id9x -.-> |"sync"|idb
id9a --> |"custom target"|id9h
id9a --> |"supported hooks"|id10a