OpenEBS cStor Discussions Notes === :::info - **Location:** https://meet.google.com/tdk-rqwj-ujp - **Date:** Every week from 12:45pm to 01:15pm on Thursday (IST) - **Agenda** 1. Walk through the status update 2. Discussion topic *Add your request, question or suggestion to our [issue list](https://github.com/openebs/openebs/issues)* *If you have anything you’d like to put on the agenda, please do so below for the next meeting:* ::: ## December 10th, 2020 Attendees: Sathya,Prateek, Shubham Discussion Notes: ## Status Updates - [name=Sathya] - Refactor the Backup and restore test case - segregate it into different test case [done] - Include Backup and Restore test case in Migration pipeline (In progress) ## December 3rd, 2020 Attendees: Sathya,Prateek, Shubham Discussion Notes: ## Status Updates - [name=Sathya] - Refactor the Backup and restore test case - segregate it into different test case [in progress] - Include Backup and Restore test case in Migration pipeline (pending) - Modify the utils to deprovision the spc/cspc pools to check the condition about volumes (In progress) - [name=Prateek] - CSI core CRDs V1 and cleanup CSINodeinfo crds (not needed) - CStor operators charts reviews - Admission webhook docs - [name=Shubham] - Upgrade CRDs V1 (will discuss with Sagar about the changes) [?] - Handle PVC delete admission to validate only for cstor PVCs - Cleanup pending bdcs on cspc deletion ## November 26th, 2020 Attendees: Sathya Discussion Notes: ## Status Updates - [name=Sathya] - Refactor the Backup and restore test case - segregate it into different test case [pending] ## November 5th, 2020 Attendees: Sathya, Ashutosh, Prateek, Shubham Discussion Notes: - If the new exporter is ready -- a change in pool deployment builder file needs to be changed in cstor-operators. ## Status Updates - [name=Sathya] - Refactor the Backup and restore test case - segregate it into different test case [pending] - Automate the test case for blockdevice replcaement when the bd is already in use by any pool [in review] - [name=Ashutosh] - Working on Review comment for NDM charts. [WIP/Under Review] - NDM Charts ci PR got approved. [approved] - Working on cstor charts to incorporate review comments.[WIP/Under Review] - [name=Prateek] - Upgrade openebs crds to V1 version [under review] - Cstor csi and zfs-localpv multi arch [under review] - Reviews in upgrade CR migration status changes - [name=Shubham] - Multi-arch images for cstor [under review] - Exporter migration to new repo ## October 29th, 2020 Attendees: Ashutosh, Prateek, Shubham, Sathya Discussion Notes: ## Status Updates - [name=Ashutosh] - Working on review comments on helm charts by David and Akhil. - Helm chart CI needs to be solved via some pre hooks. Need some research here. - [name=Shubham] - Migration CR status report [approved by shovan] - Multi-arch images for cstor [under review] - [name=Prateek] - Upgrade openebs crds to V1 version [started] - Reviews in upgrade CR migration status changes - [name=Sathya] - Automate the test case to perform openebs provision other than the openebs namespace [Done] - Automate the test case for pool expansion when the bd is already in use by any pool [In Review] - Automate the test case for blockdevice replcaement when the bd is already in use by any pool [Pending] - Refactor the Backup and restore test case - segregate it into different test case [pending] ## October 22th, 2020 Attendees: Ashutosh, Prateek, Shubham, Sathya, Akhil Discussion Notes: - Kinvolk locomotive - CSPC template support - Communicate about parallel execution of SPC and CSPC - VolumePolicy and StorageClass support - Admission namespace delete validation - Explore helm pre-hooks for graceful deletion - Helm delete 180 sec timeout to start delete forcefully ### Status Updates - [name=Ashutosh] - Helm chart review comments [under review] - Helm chart CI issue debugging [under review] - [name=Shubham] - Migration CR status report [under review] - Multi-arch images for cstor [under review] - [name=Prateek] - Major release module api changes [In review] - Update the dependent repo with module changes[In review] - Upgrade openebs crds to V1 version [not started] ## October 15th, 2020 Attendees: Sathya Discussion Notes: ### Status Updates - [name=Sathya] - Automate the test case to perform openebs provision other than the openebs namespace - Automate the test case for pool expansion when the bd is already in use by any pool ## October 8th, 2020 Attendees: Ashutosh, Sathya, Shubham Discussion Notes: ### Status Updates - [name=Ashutosh] - Working on Helm charts - [name=Sathya] - Refactor the deployment of cstor-operator and csi-provisioner test [Done] - Automate spc/cstor volume migration test script [Done] - Pipeline for spc/cstor volume migration [Done] - Automate the test case for pool expansion when the bd is already in use by any pool [Not Started] - [name=Shubham] - CStor muti arch is in progress. All the relevant PRs are under review.(Still Under review) - Automate virtual disk correction PR is under review. One function depends on the approach for zpool status (pod/exec or CR status).[Done] - Backup/restored upgrade & migration changes .[Done] - Add status reporting for cstor migration CR. [In-Progress] ## September 24th, 2020 Attendees: Sathya, Ashutosh, Shubham, Sai, Prateek Discussion Notes: - A typo fix in CSPC-operator log needs to be fixed. [Sathya Will Raise PR] - Update upgrade matrix with 2.2.0 in cspc-operators. [Sathya Will Raise PR] - CSP command execution OEP -- Shubham will take care of it. - Disaster Recovery Operations - When the cluster is destroyed and data is still intact disk there should be a workflow/automation to build the entire control plane and cStor CRs. - Will need raise OEP - Resources: - Issue Link: https://github.com/openebs/cstor-operators/issues/178 ### Status Updates - [name=Sathya] - Refactor the deployment of cstor-operator and csi-provisioner test [In review] - Automate spc/cstor volume migration test script [in review] - Pipeline for spc/cstor volume migration [In progress] - Automate the test case for pool expansion when the bd is already in use by any pool [Not Started] - [name=Ashutosh] - Helm charts CI anf gh-pages for NDM and cStor - Incorporated review comments for BD tag doc - Pending Integration tests [Not Started] - [name=Shubham] - CStor muti arch is in progress. All the relevant PRs are under review.(Under review) - Automate virtual disk correction PR is under review. One function depends on the approach for zpool status (pod/exec or CR status). - Backup/restored upgrade & migration changes still pending.[Waiting on backup/restore v1 migration] - Add status reporting for cstor migration CR. [Blocked on: 1.backup/restore v1 2. virtual disk correction] - Parallel execution of volume migration got merged. - [name=Prateek] - Openebs API v2 modules changes [WIP, backup/restore apis needs to be v1] - Review backup/restore API changes - Review cstor-operators backup/restore changes - [name=Sai] - Promote backup and restore to v1 version. [Ready for Review](@ashutosh, @prateek) - CSPC-Operator should able to identify the node label changes [Ready for Review](@ashutosh, @prateek) ## September 17th, 2020 Attendees: Sathya, Ashutosh, Prateek, Sai ### Status Updates - [name=Sathya] - Refactor the deployment of cstor-operator and csi-provisioner test [In progress] - Automate spc/cstor volume migration test script [Not Started] - Pipeline for spc/cstor volume migration [pending because of above task] - [name=Ashutosh] - Working on to set up gh-pages and actions ci for helm charts - Integrations tests [Not started] - BD tag documentation on cstor-operators repo - [name=Prateek] - Openebs API v2 modules changes [WIP, backup/restore apis needs to be v1] - CStor CSI Metrics docs (under review) - [name=Sai] - backup/restore to v1, added backward compatibility, working on unit tests [WIP] - Update the migration OEP with auto-correction of BD as per the discussed changes [WIP] ## September 10th, 2020 Attendees: Shubham, sathya, Ashutosh Discussion Notes: - BD tag feature in NDM and allowed-bd-tag in cStor. [e2e test cases, intergration test cases][UT is covered] Reference Document to understand: https://medium.com/@sonasingh46/disk-reservation-how-openebs-cstor-uses-this-feature-from-node-disk-manager-d4ff4f96b5d7 - Need to discuss Backup/Restore V1 migration with Sai. - [name=Sathya] - refactor cspc/cstor volume upgrade test script [in progress] - Pipeline for cspc/cstor volume upgrade [pending because of above task] - Automating the container runtime failure scenario for csi based volume [in progress] - Verify cstor csi target resoure limit [MV in progress] - [name=Shubham] - multi-arch upgrade approved by Michael - CStor muti arch is waiting on the cStor license refactor. All the relevant PRs are under review. - BD tag migration PR got merged. - [name=Ashutosh] - Allowed BD tag on CSPC and SPC got merged. - NDM charts PR got approved. Working on to figure out install dependencies. - Design document for volume replpica schedule is still in progress ## September 3rd, 2020 - [Ashutosh] - Working on ndm harts(in-progress) - Working on cstor-operators/csi charts (in-progress) - [Sai] - zfs/zpool command execute via CSP (design) (under-review) - node name changes identifcation in case of reboot(under review) - cvc-operator server changes for backup and restore v1 apis. - [Sathya] - Pipeline scripts for cspc/csi volume upgrade [ in progress] - Automate the test case for cstor app target affinity [ in progress] - [Prateek] - Volume create/delete events metrics analytics (pending) - openebs/api v2 modules changes (in-progress, holding it for backup/restore v1 migration) - Blog on metrics (in-progress) - Blog on Kasten backup/restore (in-progress) - PR reviews - [Shubham] - multi-arch builds for cstor - automate virtual disk correction while migration - migration CR for status reporting - Blog on migration of cstor resources to v1 - ToDo: Add a card for the user requested enhancement. - [Ajeet] - license check for openebs/api ## June 25th, 2020 ### Status Updates - [Sai] - Tracking Memory issue in cstor-istgt by using valgring(Not Started) - Test cases for REST end point(In-Review) - Need to start integration testing(Not Started) ## June 25th, 2020 Attendees: Sai, Shubham, sathya, Ashutosh, Prateek, Kiran, Sagar, Shvoan, Vishnu ### Discussion Topics - https://hackmd.io/uJL_kemTQuuwo7qmTR9q4A?view ## June 18th, 2020 Attendees: Sai, Shubham, Giri, sathya ### Status Updates - [sathya] - Automate the CSPC pool and cstor csi volume upgrade (in progress) - Automate the SPC to CSPC migration and non-csi to csi volume migration. - [Shubham] - Add cspc scale down validation (under review) - Add sanity travis test for cspc & csi volume upgrades (pending) - Add documentation for cspc/csi upgrade and migration (pending) - [Giri] - Create target with resource limit and requests specified and ensure that it is honored. - [Sai] - CSPI status should intact with OEP(Under review) - Tracking Memory issue in cstor-istgt by using valgring(In pending) - Test cases for REST end point. ## June 11th, 2020 Attendees: Ashutosh, Sai, Shubham, Prateek, sathya ### Status Updates - [sathya] - Pool pod chaos - cspc pool pod failure (restart the pool pod) while pool expansion is happening (done) - Manually verify the CSPC pool and cstor csi volume upgrade (in progress) - Manually verify the SPC to CSPC migration and non-csi to csi volume migration - Release Activities - [Shubham] - Migration from external provisioner to CSI volumes (apporved, testing job pod restarts) - Document snapshot migration (in progress) - Automate snapshot migration (in progress) - Unit tests for migration code (pending) - [Ashutosh] - CSPI UID fix in restore controller - Velero plugin backup PR review - Raised PR to support restore for CSI volume in velero plugin [Reviewers: Mayank, Kiran] - Backup and Restore manual testing for CSI volumes via velero. - Started on restore controller doc. (Yet to complete) - [Sai] - OEP for backup/restore (velero plugin has to be added by vitta) [Reviewers: Vitta, Mayank] - Backup and restore control plane and REST server manual testing(Done). - Review of restore PR (CSPI UID fix) - Fixing BDD failures in pipelines(Review in progress). - Unable to install packages in istgt container(Need to look into the issue). - Approved volume migration PR[Need to look back if there were any changes in logic]. - [Prateek] - admission webhook timeout failures in Knovoy clusters - Replace manual cstor crds with auto generated CRDs (WIP) [*Requires e2e to minimal testing] - Cstor API docs (merged done) - CVC controller mock and mock testing (PR raised) [Reviewer: Sai, Ashutosh] - csi volume operations docs (started) - PR reviews cstor-operators, volume and snapshot migrations ### Discussion Topics - Go for CSPC and NON GO for v1alpha1 - [High-Priority] Admission webhook failure is a blocker for CSPC v1 Go. (@prateek is helping here) ## June 4th, 2020 Attendees: Ashutosh, Sai, Shubham, Prateek, Sathya, Giri, Kiran ### Status Updates - [sathya] - Pool pod chaos - cspc cstor-pool container kill in cspc pool pod for pool status verification (done) - cspc pool pod failure (restart the pool pod) while pool expansion is happening (in review) - Manually verify the CSPC pool and cstor csi volume upgrade (in progress) - [Shubham] - Migration from external provisioner to CSI volumes (under review) - Automate migration image builds (pending) - Unit tests for migration code (pending) - Supporting PRs of migration got merged - SPC to CSPC migration PR got merged. - [Ashutosh] - Got restore PR merged - Reviews on backup/restore and migration PRs. - [Sai] - OEP for backup/restore (velero plugin has to be added by vitta) [Reviewers: Vitta, Mayank] - CVC Operator REST EndPoints(PR has two approvals waiting on prattek for aprrove and merging) - Backup controller PR got merged. - CSI driver code needs to be modified to support restore. - [Prateek] - Replace manual cstor crds with auto generated CRDs (pending) [*Requires e2e to minimal testing] - Cstor API docs (merge pending) - CVC controller mock and fake testing (WIP) - CSI tests enhancement ( merged) - Helm chart releaser and automation (merged) - PR reviews cstor-operators , upgrade and migrations ## May 28th, 2020 Attendees: Ashutosh, Sai, Shubham, Prateek, Sathya, Giri, Kiran ### Status Updates - [sathya] - Pool pod chaos - cspc cstor-pool container kill in cspc pool pod for pool status verification (in review) - cspc pool pod failure (restart the pool pod) while pool expansion is happening (in progress) - [Shubham] - Migration from SPC to CSPC (under review) - Migration from external provisioner to CSI volumes (in progress, need some clarification for approach) - [Sai] - OEP for backup/restore (In review[Addressing review comments], velero plugin has to ben added) [Reviewers: Vitta, Mayank] - CVC Operator REST EndPoints(WIP PR has raised, manual testing and unit testing needs to be added) - Backup controller (in pending, not yet started) - CSPI Status Capacity representation(API PR got merged, cstor-operaters PR requires reviews) [Reviewers: Ashutosh] - BlockDevice Replacement fix(PR needs reviews). [Reviewers: Ashutosh] [Release note required] - [Ashutosh] - Closed review on cstor-operators and api repo PRs and merged. PR related to migration is pending. - CSPC demo video - Working on restore controller. - [Kiran] [Updated by Ashutosh -- Feel free to edit/correct] - Working on installer - [Prateek] - Cstor API docs (In review, needs one rebase to include new changes) - CVC controller mock and fake testing (WIP) - Replace manual cstor crds with auto generated CRDs (pending) [*Requires e2e*] - Enhance CSI integration tests in travis/github-actions( under review) (Reviewer: payes. Kiran) - Helm chart release automation with index update changes ( under review) [Reviewer: Kiran] - PR reviews cstor-operators , upgrade and migrations - [Giri] - Create target with resource limit and requests specified and ensure that it is honored. ## May 21nd, 2020 Attendees: Ashutosh, Sai, Shubham, Prateek, Sathya, Giri, Kiran ### Status Updates - [sathya] - Pool pod chaos - cspc-mgmt container kill in cspc pool pod for pool status verification (in progress) [Sai] - OEP for backup/restore (In review, velero plugin has to ben added) - Backup controller (in pending) - CSPI fake controller test (in review) - CSPI Status Capacity representation. [Ashutosh] - PR reviews on cstor-operators and migration. - Start on restore controller v1 - [Shubham] - Migration from SPC to CSPC (under review) - Migration from external provisioner to CSI volumes (in progress) - CSPC webhook for pool downscale(pending) - Release Acitivity -- upgrade job examples and CI enhancements. - [Prateek] - Cstor API docs (In review) - CVC controller mock and fake testing (WIP) - Replace manual cstor crds with auto generated CRDs (pending) [*Requires e2e*] - Enhance CSI integration tests in travis/github-actions(WIP) - Helm chart release automation with index update changes - PR reviews cstor-operators , upgrade and migrations - [Kiran] - Helm chart for cStor - [Giri] ## May 14th, 2020 Attendees: Ashutosh, Sai, Shubham, Prateek, Sathya, Giri, Kiran ### Status Updates - [Shubham] - CSPC webhook for pool expansion (done) - CSI Volume upgrade (under review) - Migration from SPC to CSPC (in progress) - CSPC upgrade (done -- sanity test delayed till helm charts available) - CSPC webhook for pool downscale(pending) - Release Acitivity - [sathya] - change litmus books for cspc v1 provisioning (done) - modify litmus books for day2 ops cspc v1 (done) - Release activity - Pool pod chaos - pod container kill (in progress) - [Giri] - csi provsioner deployment litmus experiment to fetch the file from cstor-csi repo (done) - Litmus book to create volume snapshotclass(done) - [Prateek] - cStor Volume API Documetation (PR under review) - Cstor-operators and csi-operators 1.10.0 PR - CVC controller mock test (WIP) - Test the upgrade 1.0.0 to 1.10.0 in GKE - Release Activity - [Ashutosh] - CSPC and repo contributor documentation - Travis fail fast - Restore controller (pending) - Release Activity - [Sai] - OEP for backup/restore (In review, velero plugin has to ben added) - Backup controller (in pending) - CSPI fake controller test (in review) - Release Activity - SPC pool deletion fail(due to pool busy errors) - [Kiran] - Helm install for cstor drivers (push to next release) - Release Activity - Release docs ## May 7th, 2020 Attendees: Ashutosh, Sai, Shubham, Prateek, Sathya, Giri, Vitta, Kiran ### Status Updates - [Ashutosh] - Unit Tests - mock controller framework (in review) - Sanity Test in Travis (done). Run YAML to setup cstor and csi driver and deploy an busybox app. - API Documetation for cStor Pools (in-progress) - customizing docker files for cstor-operators(in-progress) - [Prateek] - cStor v1 CRDs auto generation (done) - Webhook validation for sane defaults (done) - Volume controller enhancements (done) - cStor Volume API Documetation (WIP) - Unit tests for Volume Controller (pending) - Volume policy reconcile (done) - docker customizations for cstor-csi repo - [Sai] - Rebuild estimates (done) - Webhook to restrict pvc deletion if snapshots exist (done) - Velero backup controller design (approved) [OEP can be raised openebs/openebs repo -- OEP raised in openebs/openebs#https://github.com/openebs/openebs/pull/3032]. - Unit tests on cspi controller (PR Under Review) - CStor-pool expansion when the underlying disk was expanded (design and implementation under review)(Pushed out of cStor GA scope) - Currently woeking on issues reported by e2e team for release - [Shubham] - CSPC upgrade (in-review -- WIP for sanity test) - CSI Volume upgrade (in progress) - cspc webhook for pool expansion (done) - cspc webhook for pool downscale(pending) - migration from SPC to CSPC (pending) - [Sathya] - Pool - disk replacement (done) - CSI Target Pod kill (done) - change litmus books for cspc v1 provisioning (in-progress) - modify litmus books for day2 ops cspc v1 - [Giri] - CSI provisioner refactoring in litmus (in-progress) - Soak Test Bed. (in-progress) - [Kiran] - Helm install for cstor drivers (pending) - [Vitta] - OEP on Backup and Restore via Velero with new cStor Operators. ### Discussions - cStor new Schema Repo and automation readiness. - [kiran] helm install and operators (in progress) - [kiran] tagged images - [prateek] yamls in respective repos, follows the convention of metadata - [ashutosh] github readme cstor-operators repo updates - [prateek] OEP on cstor csi driver updated and merged - [ashutosh] contributor guidelines for cstor-operators, api, upgrade - [giri] e2e pipelines under stable will show SPC and CSPC related tests executed - [kiran] OpenEBS docs to include references to new helm/operator yamls * Pool Create/Delete * Add Pools (Scale up) * Disk Replacement * Disk addition * Delete Pool (Scale down) (Pending) * Volume Create/Delete * Volume Snapshot and Clone * Volume Resize * Volume Replica Scaleup and Scale Down * Velero Backup and Restore (Pending) * Upgrade * Migration (Pending) * Volume Metrics * Pool Metrics * Grafana Dashboards (Pending) - [Ashutosh] cStor new Schema Demo/blog for 1.10 - Demo setup - ## April 30, 2020 Attendees: Ashutosh, Prateek, Shubam, Sathya, Sai, Giri, Kiran ### Status Updates - [Ashutosh] - Unit Tests - mock controller framework (in review) - Sanity Test in Travis (pending). Run YAML to setup cstor and csi driver and deploy an busybox app. - API Documetation for cStor Pools (pending) - CI via GitHub Actions (pushed out for now) - [Prateek] - cStor v1 CRDs auto generation (in review) - Webhook validation for sane defaults (in review) - Custom CSI Topology key (done) - CSI Raw block volume metrics support (Approved, hold merge) - Helm Charts linting and test automation (done) - Travis build and push images for cstor-operators (done) - Volume controller enhancements (in progress) - cStor Volume API Documetation (pending) - Unit tests for Volume Controller (pending) - [Sai] - Rebuild estimates (done, v1 version under review) - Webhook to restrict pvc deletion if snapshots exist (done) - Add status conditions on CSPI (done) - Velero backup controller (under design review) - Unit tests on cspi controller (pending) - CStor-pool expansion when the underlying disk was expanded (design and implementation under review) - [Shubham] - CSPC upgrade (in review) - CSI Volume upgrade (in progress) - migration from SPC to CSPC (pending) - [Sathya] - Pool - disk replacement (in review) - CSI Target Pod kill (in progress) - - [Giri] - Node Failure on unmanaged cluster SPC and CSPC (done) - [Kiran] - Helm install for cstor drivers (pending) #### Discussion Topics - [Ashutosh] - Backlog Review: https://github.com/orgs/openebs/projects/9 - Add: Migration of CSPI when underlying disk moves to new node. - Will user run both cstor with SPC and CSPC on same cluster. - Yes. This needs to be manually tested. - In 1.10, users can install CSPC, along with default SPC and can try out. - In 1.11/2.0, when upgrades are seamless, SPC will be disabled by default and an upgrade path will be provided to CSPC. - [Giri/Sathya] - [SOAK Test Plan for OpenEBS](https://docs.google.com/document/d/1CvzpjLpH6YiDdDynEzFAi-qUIGBd0VnjFJGAV9BG_7I/edit#heading=h.dk5vnp2mr4ax) - E2E cstor scenarios https://docs.google.com/spreadsheets/d/17UrlNoN4c69Jyl3MvAbs0vXmwf6FjzMI4dYUM--TxsI/edit#gid=1604376367 - [Pending issues from cstor-control plane] - Ability to expand pool if the underlying disk got expanded. - OEP on this - https://github.com/openebs/openebs/pull/3001 - PR on this - https://github.com/openebs/cstor-operators/pull/43 - Ability to migrate CSPI to a different node in the same CSPC - OEP on this - https://github.com/openebs/openebs/pull/3113 - PR on this - https://github.com/openebs/cstor-operators/pull/167 - Ability to migrate a CVR to a different CSPI in the same CSPC [Replica Scheduler will simplify this] - Ability to migrate the CSPI in k8s auto upgrade scenario for GKE ( User : Garrett Bates ) - https://kubernetes.slack.com/archives/CUAKPFU78/p1600276868229200 - CSPC based integration test infrastructure. [Tracked as a CARD in cStor project ] - https://github.com/openebs/openebs/issues/2290 [ Replica Distribution Feature ] - https://github.com/openebs/openebs/issues/2855 [ Over-provisioning restriction for CSPC based volumes -- Depends on replica distribution ] - CSPC integration tests [ https://github.com/openebs/cstor-operators/issues/111 ] - https://github.com/openebs/cstor-operators/issues/116 - https://github.com/openebs/openebs/issues/2939 - https://github.com/openebs/maya/issues/1568 - https://github.com/openebs/openebs/issues/2930 - https://github.com/openebs/openebs/issues/2938