OpenEBS e2e Discussions Notes

  1. Walk through the status update
  2. Discussion topic

Add your request, question or suggestion to our issue list

If you have anything you’d like to put on the agenda, please do so below for the next meeting:

December, 7th 2020

Attendees: Sathya

  • Sathya

    • Refactor the Backup and restore test case - segregate it into different test case [In progress]
    • Modified the test case to cleanup the spc pool [done]
    • refactor the deprovision logic for the application deprovison test case. [done]

November, 30th 2020

Attendees: Sathya

  • Sathya

    • Refactor the Backup and restore test case - segregate it into different test case [pending]
    • Modified the test case to cleanup the spc pool
    • refactor the deprovision logic for the application deprovison test case.

November, 23rd 2020

Attendees: Sathya

  • Sathya

    • Refactor the Backup and restore test case - segregate it into different test case [pending]
    • Refactor the spc rothreshold limit testcase [flaky]

November, 9th 2020

Attendees: Sathya

  • Sathya

    • Refactor the Backup and restore test case - segregate it into different test case [pending]

November, 2Nd 2020

Attendees: Sathya

  • Sathya
    • Automate the test case for pool expansion when the bd is already in use by any pool [done]
    • Automate the test case for blockdevice replcaement when the bd is already in use by any pool [done]
    • Refactor the Backup and restore test case - segregate it into different test case [pending]

October, 26th 2020

Attendees: Sathya

  • Sathya
    • Automate the test case to perform openebs provision other than the openebs namespace [Done]
    • Automate the test case for pool expansion when the bd is already in use by any pool [In Review]
    • Automate the test case for blockdevice replcaement when the bd is already in use by any pool [In Review]
    • Refactor the Backup and restore test case - segregate it into different test case [pending]

October, 19th 2020

Attendees: Sathya

  • Sathya
    • Automate the test case to perform openebs provision other than the openebs namespace
    • Automate the test case for pool expansion when the bd is already in use by any pool

October, 12th 2020

Attendees: Sathya, Giri

  • Aman Gupta

    • Automation of e2e-test for incremental backup & restore for zfs-localpv.
    • Restore on different nodes.

  • Sathya

    • Pipeline scripts for SPC and cStor volume migration [Done]
    • Include litmuschaos experiment for target resource limit test [Done]
    • Gitlab transition from shashank
    • backup non cstor csi volume migrate spc pool restore the backup

October, 5th 2020

Attendees: Sathya, Giri

  • Sathya

    • Pipeline scripts for SPC and cStor volume migration [in review]
    • Include litmuschaos experiment for target resource limit test [in progress]
    • Gitlab transition from shashank

  • Aman Gupta

    • Completion of backup & restore experiment [e2e-test] and Add this into pipelines [in-progress]

September, 28th 2020

Attendees: Sathya, Giri, Aman

  • Shashank

    • Resolve infra related issues.
    • Montitor pipelines and fix failed cases. [in progress]

  • Sathya

    • Automate the test case for cstor spc and cstor volume migration [in review]
    • Refactor the test case for cstor operator and csi provisioner [in review]
    • Pipeline scripts for SPC and cStor volume migration [in progress]

September, 21st 2020

Attendees: Sathya, Aman, Giri

  • Aman

    • Automate the e2e test for backup and restore for zfs-localpv [in-progress]
    • Keep continue my task with e2e for backup restore with different scenraio and on different OS.

  • Sathya

    • Automate the test case for cstor spc and cstor volume migration [in progress]
    • Refactor the test case for cstor operator and csi provisioner [in progress]
    • Pipeline scripts for SPC and cStor volume migration

September ,7th 2020

Status Updates

Attendees: Sathya, Aman, Shashank, Giri, Kiran

  • shashank

    • Adding local csi based backup/restore in pipeline [in progress]

  • sathya

    • Refactoring the csi volume and cspc pool upgrade test case. [in review]
    • Include the container run time failure test scenario for csi based cstor volume into pipeline [in review]
    • Automate the test case for app target affinity for csi cstor volume [in progress]
    • Pipeline for cspc/ci upgrade [ in progress ]

  • Aman Gupta

    • validation of the zfs-operator yaml (one for all OS) [https://github.com/openebs/zfs-localpv/pull/204] ubuntu- done by Pawan , centos 7-done, centos8- done by one user (pending on me), on k3s-pending
    • After backup restart PR gets merge, will start manual testing of this feature.

August 31st, 2020

Status Updates

Attendees: Sathya, Aman, Shashank, Giri, Kiran

  • [shashank]

    • Adding local csi based backup/restore in pipeline [inprogress]
    • Deploying lab-infra bot on GCP VM [Done]

  • Aman Gupta

    • Added pod restart (one which is deployed before upgrade zfs driver) test cases in upgrade pipeline. [done]
    • Node failure scenario for zfs-localpv [automation-done, pipeline script-pending]

  • sathya

    • Automate the test case for pool creation with block device has filesystem.[done]
    • Refactoring the csi volume and cspc pool upgrade test case. [in progress]
    • Automate the test case for app target affinity for csi cstor volume [in progress]
    • Refactor the target network loss failure in pipeline [done]
    • Refactor the spc/cstor volume migration test case. [in progress]

  • giri

    • Review the Pull Requests

August 24th, 2020

Attendees: Sathya, Aman, Shashank, Giri, Kiran

Status Updates

  • shashank

    • Openshift cluster certificate rotation and cluster renewal [In progress]
    • Refactorization and verification of Jiva internal snapshot deletion. [In progress]
    • Automate local Backup/Restore for CSI volumes. [In progress]

  • sathya

    • Automate the test case for pool creation with block device has filesystem.[in review]
    • Manually verify the complete cluster shutdown scenario [MV]
    • Refactoring the csi volume and cspc pool upgrade test case. [ in progress ]
    • Automate the test case for app target affinity for csi cstor volume [ in progress]
    • Refactor the target network loss failure in pipeline [in review]
    • refactor the spc/cstor volume migration test case. [in progress]

  • Aman Gupta

    • Node failure scenario via power off the VM's on which volume is provisioned. [manually tested, e2e-script testing pending ]
    • Going through already present test cases for backup-restore and then create a test plan for zfs-localpv backup restore.
    • blog on e2e for zfs-localpv [in bucket list]

August 17th, 2020

Attendees: Sathya, Aman, Shashank, Giri, Kiran

Status Updates

  • [shashank]

    • Gitlab licence renewal.
    • Creation Openshift-4.5 pipeline cluster for next release.
    • Refactor jiva autogen snapshot script.

  • sathya

    • Automate the test case for pool creation with block device has filesystem.[in review]
    • Manually verify the complete cluster shutdown scenario [MV in progress]
    • Refactoring the csi volume and cspc pool upgrade test case. [ in progress ]
    • Refactor the target network loss failure in pipeline [incosistent in pipeline]
    • refactor the spc/cstor volume migration test case. [in progress]

  • Aman

    • For zfs-localpv main focus will be on e2e for backup and restore feature.
    • blog on e2e with zfs-localpv [14th sept 2020]

August 10th, 2020

Attendees: Sathya, Aman, Shashank, Giri, Kiran

Status Updates

  • shashank

    • Fix pipeline faiures on konvoy & Openshift.
    • Fix and troubleshoot GitLab CI failures.

  • sathya

    • create a separate folder for cstor test cases and create readme [ create backlog]
    • release activities for RC1 with openshift, konvoy pipelines [pending]
      • triggering pipelines for RC tags [openeshift,konvoy]
      • trigger pipeline for Upgrades [AWS]
      • Manually verify spc and noncsi volume migration
      • Manually verify the cstor csi upgrade.
    • Automate the test case for pool creation with block device has filesystem.[in progress]
    • Manually verify the complete cluster shutdown scenario [ in progress]
    • Upgrade pipeline scripts for cspc pool and csi volumes in konvoy. [ done ]
    • Refactor the test scripts for CSI provisioner and cStor operator deployment test case [done]
    • Automate the reusablity of block devices test case [done]
    • Include csi based test cases into openshift pipeline [done]
    • refactor the flaky test cases [done]
      • csi volume scale down
      • node failure for csi based volumes
      • cstor operator [included task to restart ndm componets]

  • aman

    • Release activities with native-k8s pipeline.
    • Hands-on with latest changes re: volume capacity roundoff [zfs]

  • giri

    • Review the Pull Requests
    • Upgrade openebs on director-onprem cluster

August 03rd, 2020

Attendees: Sathya, Aman, Shashank, Giri, Kiran

Status Updates

  • [shashank]

    • Issue regading Gitlab job getting stuck at end of build. (Done)
    • Refactor Jiva internal snapshot deletion to restart random replica along with leader (Done)
    • Include Jiva snapshot delete experiment in pipeline (in-progress)

  • [Aman]

    • issue raised, zfs-operator is not applying on rancher (centos 7) cluster. will follow the fix for that
    • almost done with cosmetic fixes and readme for zfslocalpv in test repo. Modifying the pipeline script for accomodating the changes.
    • shared mount test is running with busybox, will manualy test once with wordpress and then we can replace busybox with wordpress.

  • [sathya]

    • Upgrade pipeline scripts for cspc pool and csi volumes. [ In progress ]
    • Refactor the test scripts for CSI provisioner and cStor operator deployment test case [in review]
    • Automate the test case to validate pool deletion is failed if the pool has the volumes [ done]
    • Validate the csi provisioner changes in openshift and konvoy platform [done]
    • Refactor the flaky test case ro threshold limit failure into openshift pipelines [in progress]
    • Verify and automate the node failure scenario for csi based volumes [done]
    • Automate the reusablity of block devices test case [in review]
      • create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC
      • create LocalPV device remove the provision and reuse the bd for creating cspc
      • Use a blockdevice to create LocalPV in ext4 filesystem remove it then use it for creating PV for xfs filesystem
    • Automate the test case to verify the pool creation is failed when in already claimed bd [done]

July 27th, 2020

Attendees: Sathya, Aman, Shashank, Giri, Kiran

Status Updates

  • [shashank]

    • Bring up Openshift-4.5 [Done]
    • Adding above platform in pipeline(includes taking of cluster snapshot, revert-script modification etc..) [Done]
    • Refactor Jiva internal snapshot deletion to restart random replica along with leader. [In-progress]

  • [sathya]

    • Refactor the test scripts for CSI provisioner and cStor operator deployment test case [in review]
    • Automate the test case to validate pool deletion is failed if the pool has the volumes [ in review]
    • Validate the csi provisioner changes in openshift and konvoy platform [manual validation is in progress]
    • Refactor the flaky test case ro threshold limit failure into openshift pipelines [in progress]
    • Verify and automate the node failure scenario for csi based volumes [manual validation is in progress]
    • Automate the reusablity of block devices test case [in progress]
      • create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC [ in review]
      • create LocalPV device remove the provision and reuse the bd for creating cspc [ in progress]
      • Use a blockdevice to create LocalPV in ext4 filesystem remove it then use it for creating PV for xfs filesystem [ pending]
    • Automate the test case to verify the pool creation is failed when in already claimed bd [pending]

  • [Aman]

    • From previous week task; readme update for zfs-localpv test cases are partially done. but from cosmetic fixes side (zfs-LocalPV=>zfs-localpv) task is paused, as changes in directory path will need modification in all the plateforms for pipline script.
    • shared mount for zfspv [Automation done & merged ; pipeline script in progress]
    • Negative test case for localpv provisioning on selected device [PR in review].

July 20th, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

Status Updates

  • [shashank]

    • Use coreOS-4.4.3 as host OS for Openshift-4.5 (done)
    • Bring up Openshift-4.5 (in-progress)

  • [sathya]

    • Refactor the test scripts for CSI provisioner and cStor operator deployment test case [ in progress]
    • Refactor the flaky test case ro threshold limit failure into openshift pipelines [ in progress]
    • Refactor the test case for snapshot creation - flaky test [done]
    • Automate the reusablity of block devices test case [pending]
      • create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC
      • create LocalPV device remove the provision and reuse the bd for creating cspc
      • Use a blockdevice to create LocalPV in ext4 filesystem remove it then use it for creating PV for xfs filesystem

  • [Aman]

    • add zfspv shared mount e2e test into pipeline and make sure its validation.
    • zfspv validation on CentOS 8 based cluster.
    • Cosmetic fixes like(zfs-LocalPV ===> zfs-localpv) in e2e-test repo as discussed in zfs syncup last friday.
    • Update the master branch for e2e-nativek8s pipeline repo with READme's

  • [Giri]

    • Bring up konvoy 1.17.7 cluster For pipeline

July 13th, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

Status Updates

  • [Shashank]

    • Stablization of e2e-pipelines (includes runner fixes)
    • Bring up stable cluster of Openshift 4.2 [Done]

  • [Aman]

    • Running the native-k8s pipelines with release tags as well upgrade pipelines.
    • Shared-mount support for zfspv [Manually verified, will automate this]

  • [Giri]

    • Replace d2iq pipeline cluster with Konvoy v1.4.4

  • [sathya]

    • Release activities
      • Upgrade pipelines for RC2 tag [done]
      • Release pipelines for RC2 tag openshift [done], konvoy [pending]
      • Verify the changes for migration removed version details form cvc [in progress]
    • Manually reusability of blockdevice [ in progress ]
      • create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC
      • create LocalPV device remove the provision and reuse the bd for creating cspc
      • Use a blockdevice to create PV in ext4 filesystem remove it then use it for creating PV for xfs filesystem

July 6th, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

Status Updates

  • [ shashank ]

    • Automation Jiva internal snapshot deletion [Done]
    • Enable test jiva inerenal snapshot deletion test in pipeline [in-progress]
    • Adding csi based backup/restore job in konvoy-pipleine [Done]

  • [sathya]

    • Automate the CSPC pool and CSI volume Migration [in review]
    • Automate the Upgrade for CSPC pool and CSI volume [ done]
    • Bring up Openshift pipeline cluster for release 1.12 [done]
    • Include below test cases into konvoy pipeline [in progress]
      • csi volume target network delay
      • cspc pool pod container kill
    • Manually reusability of blockdevice [ in progress ]
      • create SPC pool and delete the spc pool and reuse the blockdevices to create CSPC
      • create LocalPV device remove the provision and reuse the bd for creating cspc
      • Use a blockdevice to create PV in ext4 filesystem remove it then use it for creating PV for xfs filesystem

  • [ Aman ]

    • E2e test for b-tree fstype for zfspv
    • E2e test for shared mount for zfspv volume

June 29th, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

Status Updates

  • [shashank]

    • Adding csi based backup/restore job in konvoy-pipleine [Under-review]
    • Adding support for backup/restore for csi volume in experiment. [Done]
    • Test cases compatibily check with velero-1.4.0 in pipelines [Done]
    • Jiva internal snapshot deletion [In-progress]

  • [sathya]

    • Automate the CSPC pool and CSI volume migration [in progress]
    • Automate the Upgrade for CSPC pool and CSI volume [in review]
    • Include below test cases into konvoy pipeline [in progress]
      • csi volume target network delay
      • cspc pool pod container kill
      • cspc poo pod failure when the pool expansion is in progress
    • Refactor the below flaky test cases
      • OpenEBS provision [refactor to run with the custom images also] [done]
      • SPC ro-threshold limit [done]
    • Bring up Openshift pipeline cluster for release 1.12

  • [Aman]

    • zfspv-localpv tests validation on centOS 7 (konvoy pipeline cluster) [Done]
    • Include below test into konvoy [Done]
      • csi volume scaleup
      • csi volume scaledown

June 22nd, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

Status Updates

  • [Shashank]

    • Backup/Restore for CSI volume into the pipeline
    • Existing Backup/Restore expermients compatibility check with velero-V1.4.0 in pipelines.
    • Experiment for Jiva internal snapshot deletion.

  • [sathya]

    • Automate the CSPC pool and CSI volume migration
    • Automate the Upgrade for CSPC pool and CSI volume
    • Refactor the below flaky test cases in pipeline
      • OpenEBS provision [refactor to run with the custom images also]
      • SPC ro threshold limit

  • [Aman]

    • Validation of zfs-localPV tests on CentOS based cluster (Konvoy-cluster with CentOS Linux 7)

June 15th, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

Status Updates

  • [sathya]
    • Automate the CSPC and CSI volume upgrade (manual verifications done)
    • Automate the SPC to CSPC and non-csi to csi volume migration (manuall verifications done)
    • Automate the test case for container kill(cstor-pool and cstor-pool-mgmt) during disk replacement (in-progress)

June 8th, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

Status Updates

  • [shashank]

    • Deletion of failed backup (in progress)
    • Addition of new backup/restore experiments in pipeline (done)

  • [sathya]

    • Automate the pool pod failure when the pool expansion is in progress (in review)
    • Automate the test case for container kill during disk replacement (in-progress)
    • Manually verify and Automate the CSPC and CSI volume upgrade

  • [Giri]

    • Review the PRs.
    • Revamp the clusters for pipelines after migrating them into different environment.

  • [Aman]

    • Complete the automation part for raw block volume support (work in progress)

June 1st, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

Status Updates

  • [shashank]

    • Restore remote backup in different namespace (Done)
    • Backup/Restore using different S3 profile (under review)
    • Deletion of failed backup (in-progress)
    • AWS infra automation for release pipeline (Done)
    • Bring up Openshift pipeline cluster for release 1.11 (in-progress)

  • [sathya]

    • Automate the pool pod failure when the pool expansion is in progress (in progress)
    • Automate the test case for container kill during disk replacement (in-progress)
    • Automate the pool pod container failure and verify the application behaviour (in review)
    • Included a upgrade test scripts for aws platform (migrte the upgrade pipeline to gke to aws) (in review)

  • [Giri]

    • Migrate the konvoy and kubeadm pipeline clusters into different hosts (Done)
    • Explore rancher architecture for setting up feature-gates (Done)
    • Litmus experiment to set resource limit and request for cstor target pods (in-progress)

  • [Aman]

    • Automation of zfs-custom-topology support is (done). Pipeline scripts PR is there, needs to be merge.
    • Same status (^^^) for e2e of restarting docker/kubelet services with zfspv test.
    • Raw block volume support. (Automation is pending, in-progress)

May 18th, 2020

Attendees: Shashank, Sathya, Giri, Aman, Kiran

Status Updates

  • [Giri]

    • Litmus experiment to set resource limit and request for cstor target pods

  • [Shashank]

    • Backup/Restore experiment to restore the backup in different namespace from remote location. (DONE)
    • Backup/Restore when CVR's are unhealthy.

  • [Aman]

    • Custom-topology support for zfspv (automation in progress)
    • Raw block volume support (Need more exploration to automate)
    • Find work-aroud and automate docker/kubelet restart infra-chaos for zfspv into pipeline (automation is stucked because of not having the root access to nodes in pipeline)

  • [sathya]

    • Automate the pool pod failure when the pool expansion is in progress
    • Automate the pool pod container failure and verify the application behaviour

May 11th, 2020

Attendees: Somesh, Shashank, Sathya, Giri, Aman, Kiran

Status Updates

  • [Somesh]

  • [Shashank]

    • Docs regarding gitlab migration.(Done)
    • Basic infra and pipeline fix.(Done)
    • Redirection to new openshift cluster for openshift pipeline.(Done)

  • [Sathya]

    • Litmus for introducing choas (pod kill) during disk replacement (Done)
    • Automated the provision of cstor operator (As part of v1 api versio) (Done)
    • Modified the openebs provision, cspc pool provision, day2ops (pool expansion, pool scaleup) as the change in cstor operator api version (Done)
    • Litmus for introducing choas (container kill) during disk replacement (in-progress)

  • [Giri]

    • Exploring the way of using pumba for chaos injection in rancher environment.(in-progress)
    • Litmus book for creating cstor csi snapshot class.(Done)
    • Litmus book for clone creation based on cstor snapshot(in-progress)

  • [Aman]

    • Upgrade-pipeline for zfs-locapv (Done & Running)
    • Added zfs-controller in HA (done)
    • Able to get grafana dashboard up & running (in-progress)
    • Custom-topology support (in-progress)
    • raw block volume support (manually done, automation in-progress)

May 4th, 2020

Attendees: Somesh, Shashank, Sathya, Giri, Aman, Kiran

Status Updates

  • [Somesh]

    • Litmus for Jiva logging ( done )
    • Litmus for clean-up job policy ( done )
    • Litmus for pre-load changes (started)
    • Setup Jiva application on workload cluster with custom jiva pool (not started)
    • Presentation on how jiva volumes work (not started)

  • [Shashank]

    • GitLab migration to native K8s. Fixed issues (done)
    • Rebuild OpenShift cluster (in-progress)
    • Review the OpenShift cluster documentation ( in-progress )

  • [Sathya]

    • Litmus for introducing choas (pod kill, container kill) during disk replacement (in-progress)
    • Automated the bulk upgrade of cstor/jiva (done)

  • [Giri]

    • Litmus experiment for performing node failure where cstor volumes are mounted. Applying the workaround to delete the node CR and check its availability. ( done )
    • Litmus experiment for provisioning local PV on selected/tagged block devices. ( done )
    • Workload cluster was recreated with ubuntu 18.04 and k8s v1.18.2

  • [Aman]

    • Automating upgrade for ZFS Local PV ( in-progress )
    • Litmus for ZFS CSI Driver HA (in-progress)

Discussion Topics

  • [Infra Updates]
    • On-prem
      • GitLab Server
      • ZFS Local PV Native K8s Pipeline
      • OpenShift stable pipeline
      • Konvoy stable pipeline
      • Director
    • GCP
      • GitLab Runners using MicroVMs (Pipeline Executor)
    • GKE
      • Upgrade pipelines
    • Packet
      • Dev pipelines on 3 different K8s versions
Select a repo