OpenEBS e2e Discussions Notes

OpenEBS e2e Discussions Notes === :::info - **Location:** https://meet.google.com/bme-vjsj-qdp - **Date:** Every week from 12:45am to 01:15pm on Monday (IST) - **Agenda** 1. Walk through the status update 3. Discussion topic *Add your request, question or suggestion to our [issue list](https://github.com/openebs/e2e-tests/issues)* *If you have anything you’d like to put on the agenda, please do so below for the next meeting:* ::: ## December, 7th 2020 Attendees: Sathya - [name=Sathya] - Refactor the Backup and restore test case - segregate it into different test case [In progress] - Modified the test case to cleanup the spc pool [done] - refactor the deprovision logic for the application deprovison test case. [done] ## November, 30th 2020 Attendees: Sathya - [name=Sathya] - Refactor the Backup and restore test case - segregate it into different test case [pending] - Modified the test case to cleanup the spc pool - refactor the deprovision logic for the application deprovison test case. ## November, 23rd 2020 Attendees: Sathya - [name=Sathya] - Refactor the Backup and restore test case - segregate it into different test case [pending] - Refactor the spc rothreshold limit testcase [flaky] ## November, 9th 2020 Attendees: Sathya - [name=Sathya] - Refactor the Backup and restore test case - segregate it into different test case [pending] ## November, 2Nd 2020 Attendees: Sathya - [name=Sathya] - Automate the test case for pool expansion when the bd is already in use by any pool [done] - Automate the test case for blockdevice replcaement when the bd is already in use by any pool [done] - Refactor the Backup and restore test case - segregate it into different test case [pending] ## October, 26th 2020 Attendees: Sathya - [name=Sathya] - Automate the test case to perform openebs provision other than the openebs namespace [Done] - Automate the test case for pool expansion when the bd is already in use by any pool [In Review] - Automate the test case for blockdevice replcaement when the bd is already in use by any pool [In Review] - Refactor the Backup and restore test case - segregate it into different test case [pending] ## October, 19th 2020 Attendees: Sathya - [name=Sathya] - Automate the test case to perform openebs provision other than the openebs namespace - Automate the test case for pool expansion when the bd is already in use by any pool ## October, 12th 2020 Attendees: Sathya, Giri - [name=Aman Gupta] - Automation of e2e-test for incremental backup & restore for zfs-localpv. - Restore on different nodes. - [name=Sathya] - Pipeline scripts for SPC and cStor volume migration [Done] - Include litmuschaos experiment for target resource limit test [Done] - Gitlab transition from shashank - backup non cstor csi volume migrate spc pool restore the backup ## October, 5th 2020 Attendees: Sathya, Giri - [name=Sathya] - Pipeline scripts for SPC and cStor volume migration [in review] - Include litmuschaos experiment for target resource limit test [in progress] - Gitlab transition from shashank - [name=Aman Gupta] - Completion of backup & restore experiment [e2e-test] and Add this into pipelines [in-progress] ## September, 28th 2020 Attendees: Sathya, Giri, Aman - [name= Shashank] - Resolve infra related issues. - Montitor pipelines and fix failed cases. [in progress] - [name=Sathya] - Automate the test case for cstor spc and cstor volume migration [in review] - Refactor the test case for cstor operator and csi provisioner [in review] - Pipeline scripts for SPC and cStor volume migration [in progress] ## September, 21st 2020 Attendees: Sathya, Aman, Giri - [name=Aman] - Automate the e2e test for backup and restore for zfs-localpv [in-progress] - Keep continue my task with e2e for backup restore with different scenraio and on different OS. - [name=Sathya] - Automate the test case for cstor spc and cstor volume migration [in progress] - Refactor the test case for cstor operator and csi provisioner [in progress] - Pipeline scripts for SPC and cStor volume migration ## September ,7th 2020 ### Status Updates Attendees: Sathya, Aman, Shashank, Giri, Kiran - [name=shashank] - Adding local csi based backup/restore in pipeline [in progress] - [name=sathya] - Refactoring the csi volume and cspc pool upgrade test case. [in review] - Include the container run time failure test scenario for csi based cstor volume into pipeline [in review] - Automate the test case for app target affinity for csi cstor volume [in progress] - Pipeline for cspc/ci upgrade [ in progress ] - [name=Aman Gupta] - validation of the zfs-operator yaml (one for all OS) [https://github.com/openebs/zfs-localpv/pull/204] ubuntu- done by Pawan , centos 7-done, centos8- done by one user (pending on me), on k3s-pending - After backup restart PR gets merge, will start manual testing of this feature. ## August 31st, 2020 ### Status Updates Attendees: Sathya, Aman, Shashank, Giri, Kiran - [shashank] - Adding local csi based backup/restore in pipeline [inprogress] - Deploying lab-infra bot on GCP VM [Done] - [name=Aman Gupta] - Added pod restart (one which is deployed before upgrade zfs driver) test cases in upgrade pipeline. [done] - Node failure scenario for zfs-localpv [automation-done, pipeline script-pending] - [name=sathya] - Automate the test case for pool creation with block device has filesystem.[done] - Refactoring the csi volume and cspc pool upgrade test case. [in progress] - Automate the test case for app target affinity for csi cstor volume [in progress] - Refactor the target network loss failure in pipeline [done] - Refactor the spc/cstor volume migration test case. [in progress] - [name=giri] - Review the Pull Requests ## August 24th, 2020 Attendees: Sathya, Aman, Shashank, Giri, Kiran ### Status Updates - [name=shashank] - Openshift cluster certificate rotation and cluster renewal [In progress] - Refactorization and verification of Jiva internal snapshot deletion. [In progress] - Automate local Backup/Restore for CSI volumes. [In progress] - [name=sathya] - Automate the test case for pool creation with block device has filesystem.[in review] - Manually verify the complete cluster shutdown scenario [MV] - Refactoring the csi volume and cspc pool upgrade test case. [ in progress ] - Automate the test case for app target affinity for csi cstor volume [ in progress] - Refactor the target network loss failure in pipeline [in review] - refactor the spc/cstor volume migration test case. [in progress] - [name=Aman Gupta] - Node failure scenario via power off the VM's on which volume is provisioned. [manually tested, e2e-script testing pending ] - Going through already present test cases for backup-restore and then create a test plan for zfs-localpv backup restore. - blog on e2e for zfs-localpv [in bucket list] ## August 17th, 2020 Attendees: Sathya, Aman, Shashank, Giri, Kiran ### Status Updates - [shashank] - Gitlab licence renewal. - Creation Openshift-4.5 pipeline cluster for next release. - Refactor jiva autogen snapshot script. - [name=sathya] - Automate the test case for pool creation with block device has filesystem.[in review] - Manually verify the complete cluster shutdown scenario [MV in progress] - Refactoring the csi volume and cspc pool upgrade test case. [ in progress ] - Refactor the target network loss failure in pipeline [incosistent in pipeline] - refactor the spc/cstor volume migration test case. [in progress] - [name=Aman] - For zfs-localpv main focus will be on e2e for backup and restore feature. - blog on e2e with zfs-localpv [14th sept 2020] ## August 10th, 2020 Attendees: Sathya, Aman, Shashank, Giri, Kiran ### Status Updates - [name=shashank] - Fix pipeline faiures on konvoy & Openshift. - Fix and troubleshoot GitLab CI failures. - [name=sathya] - create a separate folder for cstor test cases and create readme [ create backlog] - release activities for RC1 with openshift, konvoy pipelines [pending] - triggering pipelines for RC tags [openeshift,konvoy] - trigger pipeline for Upgrades [AWS] - Manually verify spc and noncsi volume migration - Manually verify the cstor csi upgrade. - Automate the test case for pool creation with block device has filesystem.[in progress] - Manually verify the complete cluster shutdown scenario [ in progress] - Upgrade pipeline scripts for cspc pool and csi volumes in konvoy. [ done ] - Refactor the test scripts for CSI provisioner and cStor operator deployment test case [done] - Automate the reusablity of block devices test case [done] - Include csi based test cases into openshift pipeline [done] - refactor the flaky test cases [done] - csi volume scale down - node failure for csi based volumes - cstor operator [included task to restart ndm componets] - [name=aman] - Release activities with native-k8s pipeline. - Hands-on with latest changes re: volume capacity roundoff [zfs] - - [name=giri] - Review the Pull Requests - Upgrade openebs on director-onprem cluster ## August 03rd, 2020 Attendees: Sathya, Aman, Shashank, Giri, Kiran ### Status Updates - [shashank] - Issue regading Gitlab job getting stuck at end of build. (Done) - Refactor Jiva internal snapshot deletion to restart random replica along with leader (Done) - Include Jiva snapshot delete experiment in pipeline (in-progress) - [Aman] - issue raised, zfs-operator is not applying on rancher (centos 7) cluster. will follow the fix for that - almost done with cosmetic fixes and readme for zfslocalpv in test repo. Modifying the pipeline script for accomodating the changes. - shared mount test is running with busybox, will manualy test once with wordpress and then we can replace busybox with wordpress. - [sathya] - Upgrade pipeline scripts for cspc pool and csi volumes. [ In progress ] - Refactor the test scripts for CSI provisioner and cStor operator deployment test case [in review] - Automate the test case to validate pool deletion is failed if the pool has the volumes [ done] - Validate the csi provisioner changes in openshift and konvoy platform [done] - Refactor the flaky test case ro threshold limit failure into openshift pipelines [in progress] - Verify and automate the node failure scenario for csi based volumes [done] - Automate the reusablity of block devices test case [in review] - create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC - create LocalPV device remove the provision and reuse the bd for creating cspc - Use a blockdevice to create LocalPV in ext4 filesystem remove it then use it for creating PV for xfs filesystem - Automate the test case to verify the pool creation is failed when in already claimed bd [done] ## July 27th, 2020 Attendees: Sathya, Aman, Shashank, Giri, Kiran ### Status Updates - [shashank] - Bring up Openshift-4.5 [Done] - Adding above platform in pipeline(includes taking of cluster snapshot, revert-script modification etc..) [Done] - Refactor Jiva internal snapshot deletion to restart random replica along with leader. [In-progress] - [sathya] - Refactor the test scripts for CSI provisioner and cStor operator deployment test case [in review] - Automate the test case to validate pool deletion is failed if the pool has the volumes [ in review] - Validate the csi provisioner changes in openshift and konvoy platform [manual validation is in progress] - Refactor the flaky test case ro threshold limit failure into openshift pipelines [in progress] - Verify and automate the node failure scenario for csi based volumes [manual validation is in progress] - Automate the reusablity of block devices test case [in progress] - create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC [ in review] - create LocalPV device remove the provision and reuse the bd for creating cspc [ in progress] - Use a blockdevice to create LocalPV in ext4 filesystem remove it then use it for creating PV for xfs filesystem [ pending] - Automate the test case to verify the pool creation is failed when in already claimed bd [pending] - [Aman] - From previous week task; readme update for zfs-localpv test cases are partially done. but from cosmetic fixes side (zfs-LocalPV=>zfs-localpv) task is paused, as changes in directory path will need modification in all the plateforms for pipline script. - shared mount for zfspv [Automation done & merged ; pipeline script in progress] - Negative test case for localpv provisioning on selected device [PR in review]. ## July 20th, 2020 Attendees: Shashank, Sathya, Giri, Aman, Kiran ### Status Updates - [shashank] - Use coreOS-4.4.3 as host OS for Openshift-4.5 (done) - Bring up Openshift-4.5 (in-progress) - [sathya] - Refactor the test scripts for CSI provisioner and cStor operator deployment test case [ in progress] - Refactor the flaky test case ro threshold limit failure into openshift pipelines [ in progress] - Refactor the test case for snapshot creation - flaky test [done] - Automate the reusablity of block devices test case [pending] - create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC - create LocalPV device remove the provision and reuse the bd for creating cspc - Use a blockdevice to create LocalPV in ext4 filesystem remove it then use it for creating PV for xfs filesystem - [Aman] - add zfspv shared mount e2e test into pipeline and make sure its validation. - zfspv validation on CentOS 8 based cluster. - Cosmetic fixes like(zfs-LocalPV ===> zfs-localpv) in e2e-test repo as discussed in zfs syncup last friday. - Update the master branch for e2e-nativek8s pipeline repo with READme's - [Giri] - Bring up konvoy 1.17.7 cluster For pipeline ## July 13th, 2020 Attendees: Shashank, Sathya, Giri, Aman, Kiran ### Status Updates - [Shashank] - Stablization of e2e-pipelines (includes runner fixes) - Bring up stable cluster of Openshift 4.2 [Done] - [Aman] - Running the native-k8s pipelines with release tags as well upgrade pipelines. - Shared-mount support for zfspv [Manually verified, will automate this] - [Giri] - Replace d2iq pipeline cluster with Konvoy v1.4.4 - [sathya] - Release activities - Upgrade pipelines for RC2 tag [done] - Release pipelines for RC2 tag openshift [done], konvoy [pending] - Verify the changes for migration removed version details form cvc [in progress] - Manually reusability of blockdevice [ in progress ] - create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC - create LocalPV device remove the provision and reuse the bd for creating cspc - Use a blockdevice to create PV in ext4 filesystem remove it then use it for creating PV for xfs filesystem ## July 6th, 2020 Attendees: Shashank, Sathya, Giri, Aman, Kiran ### Status Updates - [ shashank ] - Automation Jiva internal snapshot deletion [Done] - Enable test jiva inerenal snapshot deletion test in pipeline [in-progress] - Adding csi based backup/restore job in konvoy-pipleine [Done] - [sathya] - Automate the CSPC pool and CSI volume Migration [in review] - Automate the Upgrade for CSPC pool and CSI volume [ done] - Bring up Openshift pipeline cluster for release 1.12 [done] - Include below test cases into konvoy pipeline [in progress] - csi volume target network delay - cspc pool pod container kill - Manually reusability of blockdevice [ in progress ] - create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC - create LocalPV device remove the provision and reuse the bd for creating cspc - Use a blockdevice to create PV in ext4 filesystem remove it then use it for creating PV for xfs filesystem - [ Aman ] - E2e test for b-tree fstype for zfspv - E2e test for shared mount for zfspv volume ## June 29th, 2020 Attendees: Shashank, Sathya, Giri, Aman, Kiran ### Status Updates - [shashank] - Adding csi based backup/restore job in konvoy-pipleine [Under-review] - Adding support for backup/restore for csi volume in experiment. [Done] - Test cases compatibily check with velero-1.4.0 in pipelines [Done] - Jiva internal snapshot deletion [In-progress] - [sathya] - Automate the CSPC pool and CSI volume migration [in progress] - Automate the Upgrade for CSPC pool and CSI volume [in review] - Include below test cases into konvoy pipeline [in progress] - csi volume target network delay - cspc pool pod container kill - cspc poo pod failure when the pool expansion is in progress - Refactor the below flaky test cases - OpenEBS provision [refactor to run with the custom images also] [done] - SPC ro-threshold limit [done] - Bring up Openshift pipeline cluster for release 1.12 - [Aman] - zfspv-localpv tests validation on centOS 7 (konvoy pipeline cluster) [Done] - Include below test into konvoy [Done] - csi volume scaleup - csi volume scaledown ## June 22nd, 2020 Attendees: Shashank, Sathya, Giri, Aman, Kiran ### Status Updates - [Shashank] - Backup/Restore for CSI volume into the pipeline - Existing Backup/Restore expermients compatibility check with velero-V1.4.0 in pipelines. - Experiment for Jiva internal snapshot deletion. - [sathya] - Automate the CSPC pool and CSI volume migration - Automate the Upgrade for CSPC pool and CSI volume - Refactor the below flaky test cases in pipeline - OpenEBS provision [refactor to run with the custom images also] - SPC ro threshold limit - [Aman] - Validation of zfs-localPV tests on CentOS based cluster (Konvoy-cluster with CentOS Linux 7) ## June 15th, 2020 Attendees: Shashank, Sathya, Giri, Aman, Kiran ### Status Updates - [sathya] - Automate the CSPC and CSI volume upgrade (manual verifications done) - Automate the SPC to CSPC and non-csi to csi volume migration (manuall verifications done) - Automate the test case for container kill(cstor-pool and cstor-pool-mgmt) during disk replacement (in-progress) ## June 8th, 2020 Attendees: Shashank, Sathya, Giri, Aman, Kiran ### Status Updates - [shashank] - Deletion of failed backup (in progress) - Addition of new backup/restore experiments in pipeline (done) - [sathya] - Automate the pool pod failure when the pool expansion is in progress (in review) - Automate the test case for container kill during disk replacement (in-progress) - Manually verify and Automate the CSPC and CSI volume upgrade - [Giri] - Review the PRs. - Revamp the clusters for pipelines after migrating them into different environment. - [Aman] - Complete the automation part for raw block volume support (work in progress) ## June 1st, 2020 Attendees: Shashank, Sathya, Giri, Aman, Kiran ### Status Updates - [shashank] - Restore remote backup in different namespace (Done) - Backup/Restore using different S3 profile (under review) - Deletion of failed backup (in-progress) - AWS infra automation for release pipeline (Done) - Bring up Openshift pipeline cluster for release 1.11 (in-progress) - [sathya] - Automate the pool pod failure when the pool expansion is in progress (in progress) - Automate the test case for container kill during disk replacement (in-progress) - Automate the pool pod container failure and verify the application behaviour (in review) - Included a upgrade test scripts for aws platform (migrte the upgrade pipeline to gke to aws) (in review) - [Giri] - Migrate the konvoy and kubeadm pipeline clusters into different hosts (Done) - Explore rancher architecture for setting up feature-gates (Done) - Litmus experiment to set resource limit and request for cstor target pods (in-progress) - [Aman] - Automation of zfs-custom-topology support is (done). Pipeline scripts PR is there, needs to be merge. - Same status (^^^) for e2e of restarting docker/kubelet services with zfspv test. - Raw block volume support. (Automation is pending, in-progress) ## May 18th, 2020 Attendees: Shashank, Sathya, Giri, Aman, Kiran ### Status Updates - [Giri] - Litmus experiment to set resource limit and request for cstor target pods - [Shashank] - Backup/Restore experiment to restore the backup in different namespace from remote location. (DONE) - Backup/Restore when CVR's are unhealthy. - [Aman] - Custom-topology support for zfspv (automation in progress) - Raw block volume support (Need more exploration to automate) - Find work-aroud and automate docker/kubelet restart infra-chaos for zfspv into pipeline (automation is stucked because of not having the root access to nodes in pipeline) - [sathya] - Automate the pool pod failure when the pool expansion is in progress - Automate the pool pod container failure and verify the application behaviour ## May 11th, 2020 Attendees: Somesh, Shashank, Sathya, Giri, Aman, Kiran ### Status Updates - [Somesh] - [Shashank] - Docs regarding gitlab migration.(Done) - Basic infra and pipeline fix.(Done) - Redirection to new openshift cluster for openshift pipeline.(Done) - [Sathya] - Litmus for introducing choas (pod kill) during disk replacement (Done) - Automated the provision of cstor operator (As part of v1 api versio) (Done) - Modified the openebs provision, cspc pool provision, day2ops (pool expansion, pool scaleup) as the change in cstor operator api version (Done) - Litmus for introducing choas (container kill) during disk replacement (in-progress) - [Giri] - Exploring the way of using pumba for chaos injection in rancher environment.(in-progress) - Litmus book for creating cstor csi snapshot class.(Done) - Litmus book for clone creation based on cstor snapshot(in-progress) - [Aman] - Upgrade-pipeline for zfs-locapv (Done & Running) - Added zfs-controller in HA (done) - Able to get grafana dashboard up & running (in-progress) - Custom-topology support (in-progress) - raw block volume support (manually done, automation in-progress) ## May 4th, 2020 Attendees: Somesh, Shashank, Sathya, Giri, Aman, Kiran ### Status Updates - [Somesh] - Litmus for Jiva logging ( done ) - Litmus for clean-up job policy ( done ) - Litmus for pre-load changes (started) - Setup Jiva application on workload cluster with custom jiva pool (not started) - Presentation on how jiva volumes work (not started) - [Shashank] - GitLab migration to native K8s. Fixed issues (done) - Rebuild OpenShift cluster (in-progress) - Review the OpenShift cluster documentation ( in-progress ) - [Sathya] - Litmus for introducing choas (pod kill, container kill) during disk replacement (in-progress) - Automated the bulk upgrade of cstor/jiva (done) - [Giri] - Litmus experiment for performing node failure where cstor volumes are mounted. Applying the workaround to delete the node CR and check its availability. ( done ) - Litmus experiment for provisioning local PV on selected/tagged block devices. ( done ) - Workload cluster was recreated with ubuntu 18.04 and k8s v1.18.2 - [Aman] - Automating upgrade for ZFS Local PV ( in-progress ) - Litmus for ZFS CSI Driver HA (in-progress) ### Discussion Topics - [Infra Updates] - On-prem - GitLab Server - ZFS Local PV Native K8s Pipeline - OpenShift stable pipeline - Konvoy stable pipeline - Director - GCP - GitLab Runners using MicroVMs (Pipeline Executor) - GKE - Upgrade pipelines - Packet - Dev pipelines on 3 different K8s versions

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.