# Gaps

## Upstream

### Quality gates (PR check)
* :red_circle: Need to move existing PR check jobs from PSI jenkins to public CI (centos ci most probably)
* :red_circle: E2E PR verification should run against Che deployed on real infra (not minikube/minishift)

* **che-plugin-registry**
* :heavy_plus_sign: Build + unit tests check
* :heavy_plus_sign: End2End test
* :red_circle: No code coverage report
* **upstream theia**
* :heavy_plus_sign: in pretty good shape IMO
* **che-theia**
* :heavy_plus_sign: Build + unit tests check
* :heavy_plus_sign: End2End test
* Code coverage is generated, but not reported (kinda hidden)
* **che-machine-exec**
* :heavy_plus_sign: PR check + unit tests in Travis
* build in Centos CI
* :red_circle: No End2End test
* :red_circle: No code coverage report
* **che-plugin-registry**
* :red_circle: No End2End test
* Build & shellcehck in CircleCI
* :red_circle: No code coverage report
* **che-devfile-registry**
* :red_circle: No PR check/quality gate
* :red_circle: No code coverage report
* **che**
* :heavy_plus_sign: Build & unit tests
* :heavy_plus_sign: End2End test
* :red_circle: No code coverage report
* **che-operator**
* :heavy_plus_sign: Docker build in Github Actions
* :red_circle: No End2End test here.
* :red_circle: No code coverage report
* **che-plugin-broker**
* :heavy_plus_sign: Build & unit tests in Circle CI
* :heavy_plus_sign: Code coverage reporting
* **che-jwt-proxy**
* :heavy_plus_sign: Build & unit tests in Circte CI
* :red_circle: No code coverage report
* **chectl**
* :red_circle: Still in che-incubator org???
* :heavy_plus_sign: Bot for upgrading dependencies
* :heavy_plus_sign: Check for semantic commit messages
* :heavy_plus_sign: Build & unit test in Central CI
* Code coverage is generated, but not reported (kinda hidden)

### Upstream nightly builds
* Some components are built in [Codenvy CI](https://ci.codenvycorp.com/) (to be deprecated) - Migration needs to happen soon (in progress)
* Some components are built in [Centos CI](https://ci.centos.org/)
* :red_circle: Some of nightly E2E tests are run behind VPN (on Red Hat internal jenkins/infra)
* :red_circle: Nightly E2E are not being run on production ready clusters - they are using solutions like `minikube`, `minishift`, `oc cluster up`
* :red_circle: The way of how `che-theia` consumes upstream ([eclipse-theia](https://github.com/eclipse-theia/theia)) can break our nightly artefacts - We need an E2E quality gate either on level of `che-theia` repo, or on nightly build of `che-theia` (preventing pushing of artifacts to public - dockerhub)

### Upstream release process
* [Described in details here](https://docs.google.com/document/d/1eVbsJjOOnPvBw4TFLYlhReLjvCescsjBM3T2sPbAiJw/edit#)
* :red_circle: Lot of distinct steps, partially automated on the level of components
* Ideally, this should be one or two big pipelines doing the release with step of manually approving (clicking button) before pushing the bits to public.

## Downstream

### Productization
* :red_circle: Some components are lacking [automated upstream sync](https://docs.google.com/spreadsheets/d/1TllnofYIlA7yTn7hIa8-LojGpHR1ACk1hVP-7fg_wsY/edit#gid=1346733864)
* missing: plugin brokers (2), jwtproxy, machineexec, plugin-openshift, plugin-kubernetes, crwctl
* :red_circle: Base image of dockerfiles used in upstream and downstream differs - makes it practically impossible to automate (Alpine vs RHEL/UBI)
* https://issues.redhat.com/browse/CRW-354
* Some images have upstream rhel.Dockerfile which can be transformed via sed into something Brew can use, but not all have been set up this way (still TODO)
* :red_circle: Base images of CRW containers use specific versions of UBI or EAP images, which need to be manually synced to upstream Che repos via PR as bots are not allowed to commit trivial changes to dockerfiles in Che repos (but we CAN and DO in crw GH and pkgs.devel repos)
* Since rhel.Dockerfile is not the default in upstream builds, CVE fixes are not often applied in Che, which builds based on Alpine instead of RHEL as the project values small images / faster startup time over security
* :red_circle: CVP tests run on containers as part of build quality check are trivial unit tests like "is java available?"; should be deeper.
* Could also investigate triggering E2E tests directly from CVP: https://issues.redhat.com/browse/CRW-349, https://issues.redhat.com/browse/CRW-207
* :red_circle: Process for updating plugins/vsix files in registry is a manual "hunt for latest" or "watch upstream commits" process; not automated or synced to upstream Che or upstream-upstream plugin/extension projects
* https://issues.redhat.com/browse/CRW-402, https://issues.redhat.com/browse/CRW-544
* :red_circle: Devfiles are a fork of upstream and renamed/sorted so watching for upstream changes is manual

### Automated CI Testing in different environments

* **Automated CI Testing in a disconnected environment**
* :red_circle: Lack of knowledge & automation in provisioning such env
* **Automated CI Testing in a proxied environment**
* :red_circle: Lack of knowledge & automation in provisioning such env
* **Automated CI Testing in an unreleased OSD**
* :red_circle: No access to such env
* **Automated CI Testing in RHPDS**
* :red_circle: Lack of automation in provisioning such env
* We may need approval of stakeholders running RHPDS to use this infra for automated testing.

### Automated Interoperability Testing
This should be completely covered by our E2E pipelines utilizing UMB message buses (AFAIK all the products are sending UMB messages, which could serve as a triggers)

* :red_circle: **OpenShift**
* :red_circle: **EAP**
* :red_circle: **RHEL**
* This one would need involvement of productization (to rebuild images based on newest RHEL/UBI images)

### Test Type Automation

* **Unit Tests Automation**
* Are we doing enough? What is the coverage?
* **Integration Tests Automation**
* :red_circle: Happy path tests ONLY, need to be extended
* **Performance Testing Automation**
* :red_circle: basic load tests implemented only
* :red_circle: Not automated
* **Recovery, Failover and Fault Tolerance Test Automation**
* :red_circle: no coverage ATM
* **Day 2 Operation Test Automation**
* :red_circle: no coverage ATM

### CI System

* :red_circle:**All Jobs defined in code**
* :red_circle:**Jenkins as a Code (JCASC)**
* :red_circle:**Leverage UMB for cross team CI messaging, triggering**
* :red_circle:**Resource Ready Verification**
* :red_circle:**Working CVP Pipeline (including Happy Path E2E Tests)**

### Data Driven Releases

* **PQI Traffic Ligths**