OpenEBS e2e Discussions Notes
===
:::info
- **Location:** https://meet.google.com/bme-vjsj-qdp

- **Date:** Every week from 12:45am to 01:15pm on Monday (IST)
- **Agenda**
1. Walk through the status update
3. Discussion topic

*Add your request, question or suggestion to our [issue list](https://github.com/openebs/e2e-tests/issues)*

*If you have anything you’d like to put on the agenda, please do so below for the next meeting:*
:::

## December, 7th 2020

Attendees: Sathya

- [name=Sathya]

- Refactor the Backup and restore test case - segregate it into different test case [In progress]
- Modified the test case to cleanup the spc pool [done]
- refactor the deprovision logic for the application deprovison test case. [done]

## November, 30th 2020

Attendees: Sathya

- [name=Sathya]

- Refactor the Backup and restore test case - segregate it into different test case [pending]
- Modified the test case to cleanup the spc pool
- refactor the deprovision logic for the application deprovison test case.

## November, 23rd 2020

Attendees: Sathya

- [name=Sathya]

- Refactor the Backup and restore test case - segregate it into different test case [pending]
- Refactor the spc rothreshold limit testcase [flaky]

## November, 9th 2020

Attendees: Sathya

- [name=Sathya]

- Refactor the Backup and restore test case - segregate it into different test case [pending]

## November, 2Nd 2020

Attendees: Sathya

- [name=Sathya]
- Automate the test case for pool expansion when the bd is already in use by any pool [done]
- Automate the test case for blockdevice replcaement when the bd is already in use by any pool [done]
- Refactor the Backup and restore test case - segregate it into different test case [pending]

## October, 26th 2020

Attendees: Sathya

- [name=Sathya]
- Automate the test case to perform openebs provision other than the openebs namespace [Done]
- Automate the test case for pool expansion when the bd is already in use by any pool [In Review]
- Automate the test case for blockdevice replcaement when the bd is already in use by any pool [In Review]
- Refactor the Backup and restore test case - segregate it into different test case [pending]

## October, 19th 2020

Attendees: Sathya

- [name=Sathya]
- Automate the test case to perform openebs provision other than the openebs namespace
- Automate the test case for pool expansion when the bd is already in use by any pool

## October, 12th 2020

Attendees: Sathya, Giri

- [name=Aman Gupta]
- Automation of e2e-test for incremental backup & restore for zfs-localpv.
- Restore on different nodes.

- [name=Sathya]
- Pipeline scripts for SPC and cStor volume migration [Done]
- Include litmuschaos experiment for target resource limit test [Done]
- Gitlab transition from shashank
- backup non cstor csi volume migrate spc pool restore the backup

## October, 5th 2020

Attendees: Sathya, Giri

- [name=Sathya]
- Pipeline scripts for SPC and cStor volume migration [in review]
- Include litmuschaos experiment for target resource limit test [in progress]
- Gitlab transition from shashank

- [name=Aman Gupta]
- Completion of backup & restore experiment [e2e-test] and Add this into pipelines [in-progress]

## September, 28th 2020

Attendees: Sathya, Giri, Aman

- [name= Shashank]
- Resolve infra related issues.
- Montitor pipelines and fix failed cases. [in progress]

- [name=Sathya]
- Automate the test case for cstor spc and cstor volume migration [in review]
- Refactor the test case for cstor operator and csi provisioner [in review]
- Pipeline scripts for SPC and cStor volume migration [in progress]

## September, 21st 2020

Attendees: Sathya, Aman, Giri

- [name=Aman]
- Automate the e2e test for backup and restore for zfs-localpv [in-progress]
- Keep continue my task with e2e for backup restore with different scenraio and on different OS.

- [name=Sathya]
- Automate the test case for cstor spc and cstor volume migration [in progress]
- Refactor the test case for cstor operator and csi provisioner [in progress]
- Pipeline scripts for SPC and cStor volume migration

## September ,7th 2020

### Status Updates

Attendees: Sathya, Aman, Shashank, Giri, Kiran

- [name=shashank]
- Adding local csi based backup/restore in pipeline [in progress]

- [name=sathya]
- Refactoring the csi volume and cspc pool upgrade test case. [in review]
- Include the container run time failure test scenario for csi based cstor volume into pipeline [in review]
- Automate the test case for app target affinity for csi cstor volume [in progress]
- Pipeline for cspc/ci upgrade [ in progress ]

- [name=Aman Gupta]
- validation of the zfs-operator yaml (one for all OS) [https://github.com/openebs/zfs-localpv/pull/204] ubuntu- done by Pawan , centos 7-done, centos8- done by one user (pending on me), on k3s-pending
- After backup restart PR gets merge, will start manual testing of this feature.

## August 31st, 2020

### Status Updates

Attendees: Sathya, Aman, Shashank, Giri, Kiran
- [shashank]
- Adding local csi based backup/restore in pipeline [inprogress]
- Deploying lab-infra bot on GCP VM [Done]

- [name=Aman Gupta]
- Added pod restart (one which is deployed before upgrade zfs driver) test cases in upgrade pipeline. [done]
- Node failure scenario for zfs-localpv [automation-done, pipeline script-pending]

- [name=sathya]
- Automate the test case for pool creation with block device has filesystem.[done]
- Refactoring the csi volume and cspc pool upgrade test case. [in progress]
- Automate the test case for app target affinity for csi cstor volume [in progress]
- Refactor the target network loss failure in pipeline [done]
- Refactor the spc/cstor volume migration test case. [in progress]

- [name=giri]
- Review the Pull Requests

## August 24th, 2020
Attendees: Sathya, Aman, Shashank, Giri, Kiran

### Status Updates
- [name=shashank]
- Openshift cluster certificate rotation and cluster renewal [In progress]
- Refactorization and verification of Jiva internal snapshot deletion. [In progress]
- Automate local Backup/Restore for CSI volumes. [In progress]

- [name=sathya]
- Automate the test case for pool creation with block device has filesystem.[in review]
- Manually verify the complete cluster shutdown scenario [MV]
- Refactoring the csi volume and cspc pool upgrade test case. [ in progress ]
- Automate the test case for app target affinity for csi cstor volume [ in progress]
- Refactor the target network loss failure in pipeline [in review]
- refactor the spc/cstor volume migration test case. [in progress]

- [name=Aman Gupta]
- Node failure scenario via power off the VM's on which volume is provisioned. [manually tested, e2e-script testing pending ]
- Going through already present test cases for backup-restore and then create a test plan for zfs-localpv backup restore.
- blog on e2e for zfs-localpv [in bucket list]

## August 17th, 2020
Attendees: Sathya, Aman, Shashank, Giri, Kiran

### Status Updates
- [shashank]
- Gitlab licence renewal.
- Creation Openshift-4.5 pipeline cluster for next release.
- Refactor jiva autogen snapshot script.

- [name=sathya]
- Automate the test case for pool creation with block device has filesystem.[in review]
- Manually verify the complete cluster shutdown scenario [MV in progress]
- Refactoring the csi volume and cspc pool upgrade test case. [ in progress ]
- Refactor the target network loss failure in pipeline [incosistent in pipeline]
- refactor the spc/cstor volume migration test case. [in progress]

- [name=Aman]
- For zfs-localpv main focus will be on e2e for backup and restore feature.
- blog on e2e with zfs-localpv [14th sept 2020]

## August 10th, 2020
Attendees: Sathya, Aman, Shashank, Giri, Kiran

### Status Updates
- [name=shashank]

- Fix pipeline faiures on konvoy & Openshift.
- Fix and troubleshoot GitLab CI failures.

- [name=sathya]
- create a separate folder for cstor test cases and create readme [ create backlog]
- release activities for RC1 with openshift, konvoy pipelines [pending]
- triggering pipelines for RC tags [openeshift,konvoy]
- trigger pipeline for Upgrades [AWS]
- Manually verify spc and noncsi volume migration
- Manually verify the cstor csi upgrade.
- Automate the test case for pool creation with block device has filesystem.[in progress]
- Manually verify the complete cluster shutdown scenario [ in progress]
- Upgrade pipeline scripts for cspc pool and csi volumes in konvoy. [ done ]
- Refactor the test scripts for CSI provisioner and cStor operator deployment test case [done]
- Automate the reusablity of block devices test case [done]
- Include csi based test cases into openshift pipeline [done]
- refactor the flaky test cases [done]
- csi volume scale down
- node failure for csi based volumes
- cstor operator [included task to restart ndm componets]

- [name=aman]
- Release activities with native-k8s pipeline.
- Hands-on with latest changes re: volume capacity roundoff [zfs]
-

- [name=giri]
- Review the Pull Requests
- Upgrade openebs on director-onprem cluster

## August 03rd, 2020
Attendees: Sathya, Aman, Shashank, Giri, Kiran
### Status Updates

- [shashank]
- Issue regading Gitlab job getting stuck at end of build. (Done)
- Refactor Jiva internal snapshot deletion to restart random replica along with leader (Done)
- Include Jiva snapshot delete experiment in pipeline (in-progress)
- [Aman]
- issue raised, zfs-operator is not applying on rancher (centos 7) cluster. will follow the fix for that
- almost done with cosmetic fixes and readme for zfslocalpv in test repo. Modifying the pipeline script for accomodating the changes.
- shared mount test is running with busybox, will manualy test once with wordpress and then we can replace busybox with wordpress.

- [sathya]
- Upgrade pipeline scripts for cspc pool and csi volumes. [ In progress ]
- Refactor the test scripts for CSI provisioner and cStor operator deployment test case [in review]
- Automate the test case to validate pool deletion is failed if the pool has the volumes [ done]
- Validate the csi provisioner changes in openshift and konvoy platform [done]
- Refactor the flaky test case ro threshold limit failure into openshift pipelines [in progress]
- Verify and automate the node failure scenario for csi based volumes [done]
- Automate the reusablity of block devices test case [in review]
- create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC
- create LocalPV device remove the provision and reuse the bd for creating cspc
- Use a blockdevice to create LocalPV in ext4 filesystem remove it then use it for creating PV for xfs filesystem
- Automate the test case to verify the pool creation is failed when in already claimed bd [done]

## July 27th, 2020
Attendees: Sathya, Aman, Shashank, Giri, Kiran

### Status Updates
- [shashank]
- Bring up Openshift-4.5 [Done]
- Adding above platform in pipeline(includes taking of cluster snapshot, revert-script modification etc..) [Done]
- Refactor Jiva internal snapshot deletion to restart random replica along with leader. [In-progress]

- [sathya]
- Refactor the test scripts for CSI provisioner and cStor operator deployment test case [in review]
- Automate the test case to validate pool deletion is failed if the pool has the volumes [ in review]
- Validate the csi provisioner changes in openshift and konvoy platform [manual validation is in progress]
- Refactor the flaky test case ro threshold limit failure into openshift pipelines [in progress]
- Verify and automate the node failure scenario for csi based volumes [manual validation is in progress]
- Automate the reusablity of block devices test case [in progress]
- create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC [ in review]
- create LocalPV device remove the provision and reuse the bd for creating cspc [ in progress]
- Use a blockdevice to create LocalPV in ext4 filesystem remove it then use it for creating PV for xfs filesystem [ pending]
- Automate the test case to verify the pool creation is failed when in already claimed bd [pending]

- [Aman]
- From previous week task; readme update for zfs-localpv test cases are partially done. but from cosmetic fixes side (zfs-LocalPV=>zfs-localpv) task is paused, as changes in directory path will need modification in all the plateforms for pipline script.
- shared mount for zfspv [Automation done & merged ; pipeline script in progress]
- Negative test case for localpv provisioning on selected device [PR in review].

## July 20th, 2020
Attendees: Shashank, Sathya, Giri, Aman, Kiran

### Status Updates
- [shashank]
- Use coreOS-4.4.3 as host OS for Openshift-4.5 (done)
- Bring up Openshift-4.5 (in-progress)

- [sathya]
- Refactor the test scripts for CSI provisioner and cStor operator deployment test case [ in progress]
- Refactor the flaky test case ro threshold limit failure into openshift pipelines [ in progress]
- Refactor the test case for snapshot creation - flaky test [done]
- Automate the reusablity of block devices test case [pending]
- create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC
- create LocalPV device remove the provision and reuse the bd for creating cspc
- Use a blockdevice to create LocalPV in ext4 filesystem remove it then use it for creating PV for xfs filesystem

- [Aman]
- add zfspv shared mount e2e test into pipeline and make sure its validation.
- zfspv validation on CentOS 8 based cluster.
- Cosmetic fixes like(zfs-LocalPV ===> zfs-localpv) in e2e-test repo as discussed in zfs syncup last friday.
- Update the master branch for e2e-nativek8s pipeline repo with READme's

- [Giri]

- Bring up konvoy 1.17.7 cluster For pipeline

## July 13th, 2020
Attendees: Shashank, Sathya, Giri, Aman, Kiran

### Status Updates
- [Shashank]
- Stablization of e2e-pipelines (includes runner fixes)
- Bring up stable cluster of Openshift 4.2 [Done]

- [Aman]
- Running the native-k8s pipelines with release tags as well upgrade pipelines.
- Shared-mount support for zfspv [Manually verified, will automate this]

- [Giri]
- Replace d2iq pipeline cluster with Konvoy v1.4.4

- [sathya]
- Release activities
- Upgrade pipelines for RC2 tag [done]
- Release pipelines for RC2 tag openshift [done], konvoy [pending]
- Verify the changes for migration removed version details form cvc [in progress]
- Manually reusability of blockdevice [ in progress ]
- create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC
- create LocalPV device remove the provision and reuse the bd for creating cspc
- Use a blockdevice to create PV in ext4 filesystem remove it then use it for creating PV for xfs filesystem

## July 6th, 2020
Attendees: Shashank, Sathya, Giri, Aman, Kiran

### Status Updates
- [ shashank ]
- Automation Jiva internal snapshot deletion [Done]
- Enable test jiva inerenal snapshot deletion test in pipeline [in-progress]
- Adding csi based backup/restore job in konvoy-pipleine [Done]

- [sathya]
- Automate the CSPC pool and CSI volume Migration [in review]
- Automate the Upgrade for CSPC pool and CSI volume [ done]
- Bring up Openshift pipeline cluster for release 1.12 [done]
- Include below test cases into konvoy pipeline [in progress]
- csi volume target network delay
- cspc pool pod container kill
- Manually reusability of blockdevice [ in progress ]
- create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC
- create LocalPV device remove the provision and reuse the bd for creating cspc
- Use a blockdevice to create PV in ext4 filesystem remove it then use it for creating PV for xfs filesystem

- [ Aman ]
- E2e test for b-tree fstype for zfspv
- E2e test for shared mount for zfspv volume

## June 29th, 2020
Attendees: Shashank, Sathya, Giri, Aman, Kiran

### Status Updates
- [shashank]
- Adding csi based backup/restore job in konvoy-pipleine [Under-review]
- Adding support for backup/restore for csi volume in experiment. [Done]
- Test cases compatibily check with velero-1.4.0 in pipelines [Done]
- Jiva internal snapshot deletion [In-progress]

- [sathya]
- Automate the CSPC pool and CSI volume migration [in progress]
- Automate the Upgrade for CSPC pool and CSI volume [in review]
- Include below test cases into konvoy pipeline [in progress]
- csi volume target network delay
- cspc pool pod container kill
- cspc poo pod failure when the pool expansion is in progress
- Refactor the below flaky test cases
- OpenEBS provision [refactor to run with the custom images also] [done]
- SPC ro-threshold limit [done]
- Bring up Openshift pipeline cluster for release 1.12

- [Aman]
- zfspv-localpv tests validation on centOS 7 (konvoy pipeline cluster) [Done]
- Include below test into konvoy [Done]
- csi volume scaleup
- csi volume scaledown

## June 22nd, 2020
Attendees: Shashank, Sathya, Giri, Aman, Kiran

### Status Updates
- [Shashank]
- Backup/Restore for CSI volume into the pipeline
- Existing Backup/Restore expermients compatibility check with velero-V1.4.0 in pipelines.
- Experiment for Jiva internal snapshot deletion.

- [sathya]
- Automate the CSPC pool and CSI volume migration
- Automate the Upgrade for CSPC pool and CSI volume
- Refactor the below flaky test cases in pipeline
- OpenEBS provision [refactor to run with the custom images also]
- SPC ro threshold limit

- [Aman]
- Validation of zfs-localPV tests on CentOS based cluster (Konvoy-cluster with CentOS Linux 7)

## June 15th, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

### Status Updates

- [sathya]
- Automate the CSPC and CSI volume upgrade (manual verifications done)
- Automate the SPC to CSPC and non-csi to csi volume migration (manuall verifications done)
- Automate the test case for container kill(cstor-pool and cstor-pool-mgmt) during disk replacement (in-progress)

## June 8th, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

### Status Updates

- [shashank]
- Deletion of failed backup (in progress)
- Addition of new backup/restore experiments in pipeline (done)

- [sathya]
- Automate the pool pod failure when the pool expansion is in progress (in review)
- Automate the test case for container kill during disk replacement (in-progress)
- Manually verify and Automate the CSPC and CSI volume upgrade

- [Giri]
- Review the PRs.
- Revamp the clusters for pipelines after migrating them into different environment.

- [Aman]
- Complete the automation part for raw block volume support (work in progress)

## June 1st, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

### Status Updates

- [shashank]
- Restore remote backup in different namespace (Done)
- Backup/Restore using different S3 profile (under review)
- Deletion of failed backup (in-progress)
- AWS infra automation for release pipeline (Done)
- Bring up Openshift pipeline cluster for release 1.11 (in-progress)

- [sathya]
- Automate the pool pod failure when the pool expansion is in progress (in progress)
- Automate the test case for container kill during disk replacement (in-progress)
- Automate the pool pod container failure and verify the application behaviour (in review)
- Included a upgrade test scripts for aws platform (migrte the upgrade pipeline to gke to aws) (in review)

- [Giri]
- Migrate the konvoy and kubeadm pipeline clusters into different hosts (Done)
- Explore rancher architecture for setting up feature-gates (Done)
- Litmus experiment to set resource limit and request for cstor target pods (in-progress)

- [Aman]
- Automation of zfs-custom-topology support is (done). Pipeline scripts PR is there, needs to be merge.
- Same status (^^^) for e2e of restarting docker/kubelet services with zfspv test.
- Raw block volume support. (Automation is pending, in-progress)

## May 18th, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

### Status Updates

- [Giri]

- Litmus experiment to set resource limit and request for cstor target pods

- [Shashank]

- Backup/Restore experiment to restore the backup in different namespace from remote location. (DONE)
- Backup/Restore when CVR's are unhealthy.

- [Aman]
- Custom-topology support for zfspv (automation in progress)
- Raw block volume support (Need more exploration to automate)
- Find work-aroud and automate docker/kubelet restart infra-chaos for zfspv into pipeline (automation is stucked because of not having the root access to nodes in pipeline)

- [sathya]

- Automate the pool pod failure when the pool expansion is in progress
- Automate the pool pod container failure and verify the application behaviour

## May 11th, 2020

Attendees: Somesh, Shashank, Sathya, Giri, Aman, Kiran

### Status Updates

- [Somesh]

- [Shashank]
- Docs regarding gitlab migration.(Done)
- Basic infra and pipeline fix.(Done)
- Redirection to new openshift cluster for openshift pipeline.(Done)

- [Sathya]
- Litmus for introducing choas (pod kill) during disk replacement (Done)
- Automated the provision of cstor operator (As part of v1 api versio) (Done)
- Modified the openebs provision, cspc pool provision, day2ops (pool expansion, pool scaleup) as the change in cstor operator api version (Done)
- Litmus for introducing choas (container kill) during disk replacement (in-progress)

- [Giri]
- Exploring the way of using pumba for chaos injection in rancher environment.(in-progress)
- Litmus book for creating cstor csi snapshot class.(Done)
- Litmus book for clone creation based on cstor snapshot(in-progress)

- [Aman]
- Upgrade-pipeline for zfs-locapv (Done & Running)
- Added zfs-controller in HA (done)
- Able to get grafana dashboard up & running (in-progress)
- Custom-topology support (in-progress)
- raw block volume support (manually done, automation in-progress)

## May 4th, 2020

Attendees: Somesh, Shashank, Sathya, Giri, Aman, Kiran

### Status Updates

- [Somesh]
- Litmus for Jiva logging ( done )
- Litmus for clean-up job policy ( done )
- Litmus for pre-load changes (started)
- Setup Jiva application on workload cluster with custom jiva pool (not started)
- Presentation on how jiva volumes work (not started)

- [Shashank]
- GitLab migration to native K8s. Fixed issues (done)
- Rebuild OpenShift cluster (in-progress)
- Review the OpenShift cluster documentation ( in-progress )

- [Sathya]
- Litmus for introducing choas (pod kill, container kill) during disk replacement (in-progress)
- Automated the bulk upgrade of cstor/jiva (done)

- [Giri]
- Litmus experiment for performing node failure where cstor volumes are mounted. Applying the workaround to delete the node CR and check its availability. ( done )
- Litmus experiment for provisioning local PV on selected/tagged block devices. ( done )
- Workload cluster was recreated with ubuntu 18.04 and k8s v1.18.2

- [Aman]
- Automating upgrade for ZFS Local PV ( in-progress )
- Litmus for ZFS CSI Driver HA (in-progress)

### Discussion Topics

- [Infra Updates]
- On-prem
- GitLab Server
- ZFS Local PV Native K8s Pipeline
- OpenShift stable pipeline
- Konvoy stable pipeline
- Director
- GCP
- GitLab Runners using MicroVMs (Pipeline Executor)
- GKE
- Upgrade pipelines
- Packet
- Dev pipelines on 3 different K8s versions