owned this note
owned this note
Published
Linked with GitHub
# Velero Community Meeting and Open Discussion Notes
_This doc is meant to serve as the "one doc to rule them all" for Velero community meetings and open discussions._
_Join our [Google Group](https://groups.google.com/forum/#!forum/projectvelero) to get updates on the project and invites to community meetings._
_**Community meeting schedule**:_
- Bi-weekly community meeting alternating every week between Beijing Friendly timezone and EST/Europe Friendly Timezone (See [calendar](https://calendar.google.com/calendar/embed?src=6e41fca3d22f24b2890fdbfc6b17f75dd97885a8d9ed9dab642355bdee598add@group.calendar.google.com))
- Join us at [Zoom Link](https://broadcom.zoom.us/j/94416678753?pwd=YkptN1k4M2lrUTdGbitNTmorODcvUT09)
- Beijing/US friendly - we start at 8am Beijing Time(bound to CST) / 8pm EDT(7pm EST) / 5pm PDT(4pm PST) / 2am CEST(1am CET) - [Convert to your time zone](http://www.thetimezoneconverter.com/?t=09:00&tz=PT%20%28Pacific%20Time%29)
- US/Europe friendly - we start at 10am ET(bound to ET) / 7am PDT(7am PST) / 4pm CEST (3pm CET) / 10pm/11pm CST - [Convert to your time zone](http://www.thetimezoneconverter.com/?t=17:00&tz=PT%20%28Pacific%20Time%29)
_You can find recordings of these meeting [on YouTube](https://www.youtube.com/playlist?list=PL7bmigfV0EqQRysvqvqOtRNk4L5S7uqwM)._
Archives of older meeting notes
* https://hackmd.io/@jonasrosland/SyV22skpL - Archive 2019
* https://hackmd.io/ukRqw4IxRNeXFwoYmp4iJQ - Archive 2020
* [Achived notes 2021](https://github.com/vmware-tanzu/velero/wiki/Velero-Community-Meeting-and-Open-Discussion-Notes--2021)
* https://hackmd.io/or-hIzhbRwaB-vjsxs-qnw - Archive 2022
* https://hackmd.io/Jq6F5zqZR7S80CeDWUklkA - Archive 2023
Add your request, question or suggestion to [our issue list](https://github.com/vmware-tanzu/velero/issues)
_If you have anything you'd like to put on the agenda, please do so below for the next meeting:_
## RoadMap - on-going
Link: https://hackmd.io/iQy7XmPOT0y8lrCuAFL4PQ
# Jul 23, 2024 (US/Europe)
### Attendees
- Scott Seago (Red Hat)
- Shubham Pampattiwar (Red Hat)
- Mateus Oliveira (Red Hat)
- Tiger Kaovilai (Red Hat)
- Michal Pryc (Red Hat)
- Daniel Jiang (Broadcom)
### Status
1.14.1: tentative release on 26th Aug (delayed). Issues https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+label%3Atarget%2F1.14.1
1.15: feature freeze on 28th, candidates https://github.com/vmware-tanzu/velero/issues?q=is%3Aopen+is%3Aissue+label%3A1.15-candidate
- Scott
- Backup perf improvements work
- Design PR merged (thanks for the reviews!)
- First (of 4) implementation PRs is for the ItemBlockAction plugin type
- https://github.com/vmware-tanzu/velero/pull/8026 (ready for review)
- Next PR will be implementing IBA plugins corresponding to the intermal BIA plugins (PVC for Pod, PV for PVC, etc.)
- Shubham
- Velero 1.14.1 issues
- PR reviews
- Tiger
- Designs
- Velero Download API
- Retry patch on restore
### Discussion topics
- Daniel
- https://github.com/vmware-tanzu/velero/issues/8018
- Do we wanna revert the change?
# Jul 17, 2024 (China/US)
### Attendees
- Yonghui Li (Broadcom)
- Yunqian Chen (Dell)
- Xun Jiang (Broadcom)
- Jiaolin Yang (Broadcom)
- Shubham Pampattiwar (Red Hat)
- Scott Seago (Red Hat)
- Wenkai Yin (Broadcom)
### Status
1.14.1: tentative release on 26th Aug (delayed). Issues https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+label%3Atarget%2F1.14.1
1.15: feature freeze on 28th, candidates https://github.com/vmware-tanzu/velero/issues?q=is%3Aopen+is%3Aissue+label%3A1.15-candidate
- Yonghui
- Data mover micro service tasks
- Need help on code review, need one more reviewer
- Scott and Shubham will help
- Backup repository configuration design https://github.com/vmware-tanzu/velero/pull/7963
- Backup PVC configuration design https://github.com/vmware-tanzu/velero/pull/7982
- Scott
- Backup perf improvements work
- https://github.com/vmware-tanzu/velero/pull/7628
- Design PR has 3 acks, and all feedback is responded to ready to merge?
- I started phase 1 impl work today.
- Shubham
- Backup is marked as waitingForPluginOperationsPartiallyfailed when VolumeSnapshotContent has an error
- Issue link: https://github.com/vmware-tanzu/velero/issues/7356
- PR: https://github.com/vmware-tanzu/velero/pull/8023
- Expose PVPatchMaximumDuration timeout for custom configuration
- Issue link: https://github.com/vmware-tanzu/velero/issues/8020
- Draft PR: https://github.com/vmware-tanzu/velero/pull/8021
- Xun
- Repository maintenance job configuration design
- E2E enhancement taks
- Tiger
- Velero Download API
- Studied more about ingress controllers on KinD and EKS
- PRs ready: [#7793](https://github.com/vmware-tanzu/velero/pull/7793) [#7913](https://github.com/vmware-tanzu/velero/pull/7913) [#7942](https://github.com/vmware-tanzu/velero/pull/7942)
### Discussion topics
- Tiger
- [Velero didn't retry on failed Restore CR status update, causing the CR to remain stuck in "InProgress"
#7207](https://github.com/vmware-tanzu/velero/issues/7207)
Velero had trouble connecting to the API server, but eventually, the restore was successful. After the connection was restored, it didn't attempt to update, causing the restore CR to be stuck at "In progress" indefinitely.
- we had discussed [retry](https://github.com/vmware-tanzu/velero/pull/7845) or [requeue](https://github.com/vmware-tanzu/velero/pull/7863) last time.
- suggestions:
- establish backup/restore window
- what if we don't know when api will go down?
- restart velero pod to trigger mark as failed
- other in-progress killed
- requeue
- complicates future reconciler updates (ex: [cancel](https://github.com/vmware-tanzu/velero/pull/7863#discussion_r1636304591), parallel backups)
- retry until patch is successful - simpler vs requeue
- reuse existing configurable timeouts
- create design to cover UX changes
# Jul 9, 2024 (US/Europe)
### Attendees
- Mateus Oliveira (Red Hat)
- Scott Seago (Red Hat)
- Michal Pryc (Red Hat)
- Wes Hayutin (Red Hat)
- Tiger Kaovilai (Red Hat)
- Anshul Ahuja (Microsoft)
- Shubham Pampattiwar (Red Hat)
- Yonghui Li (Broadcom)
### Status
- Tiger
- Slack
- Design Velero Server Download API
### Discussion topics
- [Michal] Design for backup repository configurations proposed by Lyndon-Li: https://github.com/vmware-tanzu/velero/pull/7963
- Scott
- Requesting final reviews on backup performance improvements, since Phase 1 needs to be implemented in Velero 1.15
- https://github.com/vmware-tanzu/velero/pull/7628
- Plugin memory leak
- https://github.com/vmware-tanzu/velero/issues/7925
- Initially reported on Azure, but we're seeing it with AWS as well
- Shubham
- Requesting reviews on [Skip PV patch step in Restore workflow for WaitForFirstConsumer VolumeBindingMode Pending state PVCs](https://github.com/vmware-tanzu/velero/pull/7953)
# Jul 3, 2024 (China/US)
### Attendees
- Yonghui Li (Broadcom)
- Xun Jiang (Broadcom)
- Shubham Pampattiwar (Red Hat)
- Scott Seago (Red Hat)
- Tiger Kaovilai (Red Hat)
- Jiaolin Yang (Broadcom)
- Wenkai Yin (Broadcom)
### Status
1.14.1: tentative release on 5th Aug. Issues https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+label%3Atarget%2F1.14.1
1.15: feature freeze on 28th, candidates https://github.com/vmware-tanzu/velero/issues?q=is%3Aopen+is%3Aissue+label%3A1.15-candidate
- Yonghui
- Data mover micro service tasks
- Backup repository configuration design https://github.com/vmware-tanzu/velero/pull/7963
- Shubham
- Updated PR [Skip PV patch step in Restore workflow for WaitForFirstConsumer VolumeBindingMode Pending state PVCs](https://github.com/vmware-tanzu/velero/pull/7953)
- PR reviews
- Tiger
- [Enable SDKv2 s3 debug logs](https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/207)
- [Design: Download server for Velero client](https://github.com/vmware-tanzu/velero/issues/7432)
- Xun
- E2E enhencement
- 1.15 and 1.14.1 issues fix
### Discussion topics
- [Xun]
- [issue: How to handle non-existing namespace in the backup namespace filters](https://github.com/vmware-tanzu/velero/issues/7928)
- [proposal PR 7695](https://github.com/vmware-tanzu/velero/pull/7965)
- Scott
- Requesting final reviews on backup performance improvements, since Phase 1 needs to be implemented in Velero 1.15
- https://github.com/vmware-tanzu/velero/pull/7628
# Jun 25, 2024 (US/Europe)
### Attendees
- Scott Seago (Red Hat)
- Michal Pryc (Red Hat)
- Anshul Ahuja (Microsoft)
- Mateus Oliveira (Red Hat)
- Shubham Pampattiwar (Red Hat)
- Tiger Kaovilai(Red Hat)
- Wes Hayutin (Red Hat)
### Status
1.14: GA on 17th. Some 1.14.1 issues https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+label%3Atarget%2F1.14.1
1.15: Collecting requirements, candidates reviewing https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+label%3A1.15-candidate
- Scott
- https://github.com/vmware-tanzu/velero/pull/7899
- Fix for unschedulable pods for DM
- Shubham
- Will update PRs for 1.4.1 issues
- https://github.com/vmware-tanzu/velero/issues/7866
- https://github.com/vmware-tanzu/velero/issues/7356
### Discussion topics
- [Anshul] https://github.com/vmware-tanzu/velero/pull/7924
- [Michal] https://github.com/vmware-tanzu/velero/issues/7620
- [Wes] - Deprecation Policy https://github.com/vmware-tanzu/velero/pull/5532
- [Wes] - upcoming k8s features - VolumeSnapshotGroups && CSI block CBT. Any discussion required in upcoming meetings?
# Jun 19, 2024 (China/US)
### Attendees
- Yonghui Li (Broadcom)
- Daniel Jiang (Broadcom)
- Scott Seago (Red Hat)
- Xun Jiang (Broadcom)
- Shubham Pampattiwar (Red Hat)
- Wes Hayutin (Red Hat)
- Tiger Kaovilai (Red Hat)
- Jiaolin Yang (Broadcom)
### Status
1.14: GA on 17th. Some 1.14.1 issues https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+label%3Atarget%2F1.14.1
1.15: Collecting requirements, candidates reviewing https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+label%3A1.15-candidate
- Yonghui
- Data mover micro service design https://github.com/vmware-tanzu/velero/pull/7576
- Issue https://github.com/vmware-tanzu/velero/issues/7896
- Daniel
- Fixed the bug in RC2, Released v1.14.0
- Xun
- Worked on v1.14.0 RC and GA tasks
### Discussion topics
- Yonghui
- About the Changing PVC selected-node feature https://velero.io/docs/v1.14/restore-reference/#changing-pvc-selected-node. https://github.com/vmware-tanzu/velero/issues/7890. Deprecate or enahnce? --- document it only works for Immediate volumes in 1.14.1; deprecate it in 1.15 and follow the deprecation policy
- Daniel
- Discuss the timeline for v1.15.0 https://github.com/vmware-tanzu/velero/wiki/1.15-Roadmap
- Scott
- Problems with DM "fail early"
- https://github.com/vmware-tanzu/velero/issues/7898
- Certain conditions are considered "unrecoverable" by velero:
- ImagePullBackoff
- Failed or Unknown pod phase
- Pending pod with "unschedulable" condition
- The "unschedulable" condition isn't necessarily "unrecoverable". In fact, it's a normal condition of the pod while the PVC isn't yet bound to the PV during provisoning phase. Most Storage Classes seem to work fine here, but if the storage drive takes longer to provision PVs, then velero may end up canceling on the DU/DD before the provisioner has time to provision and bind the PVC. We're seeing this right now with Ceph volumes
- Two possible solutions:
- Don't consider "unschedulable"/Pending unrecoverable. This allows provisioning to complete normally, even when slow. Downside is if "unschedulable" isn't resolved (i.e. in cases where there's no node available), then we take 30 minutes to time out. Linked draft PR implements this approach
- Still consider "unschedulable" to be unrecoverable, but only after polling until timeout -- a shorter timeout than the overall DM 30 minutes, probably more like 2 minutes.
# Jun 11, 2024 (US/Europe)
### Attendees
- Scott Seago (Red Hat)
- Raghuram Devarakonda (CloudCasa)
- Mateus Oliveira (Red Hat)
- Shubham Pampattiwar (Red Hat)
- Tiger Kaovilai (Red Hat)
- Michal Pryc (Red Hat)
- Anshul Ahuja (Microsoft)
- Wes Hayutin (Red Hat)
### Status
1.14: RC2 was created
1.15: Collecting requirements, candidates https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+label%3A1.15-candidate
- Shubham:
- Created PR for [Add Restore err in Finalizing PV Patch phase if SC VolumeBindingMode is WaitForFirstConsumer](https://github.com/vmware-tanzu/velero/pull/7869) for issue https://github.com/vmware-tanzu/velero/issues/7866
### Discussion topics
- [API Server temporary outage Requeue/Retry patching Completion phase discussion](https://github.com/vmware-tanzu/velero/pull/7845)
- To unstuck backup status if last status patch call fails
- Yonghui will follow up
- We could store Complete/PartiallyFailed in-memory to patch next reconcile for already uploaded backups
- Retry is still potential solution
- can add in downstream?
# Jun 5, 2024 (China/US)
### Attendees
- Yonghui Li (Broadcom)
- Daniel Jiang (Broadcom)
- Xun Jiang (Broadcom)
- Scott Seago (Red Hat)
- Shubham Pampattiwar (Red Hat)
- Jiaolin Yang (Broadcom)
- Tiger Kaovilai (Red Hat)
- Wenkai Yin(Broadcom)
### Status
1.14: RC1 was created, running manually tests on issue fixes, will create RC2
1.15: Collecting requirements, candidates https://github.com/vmware-tanzu/velero/issues?q=is%3Aissue+is%3Aopen+label%3A1.15-candidate
- Yonghui
- Follow up comments/discussions on data mover micro service design https://github.com/vmware-tanzu/velero/pull/7576
- Daniel
- Fix issue 7849 (targeting v1.14 RC2)
- Prepare for RC2
- Planning v1.15
- Xun
- Test v1.14-rc.1
- Verify checked-in fixes.
- Tiger
- [Retry backup/restore completion/finalizing status patching #7845](https://github.com/vmware-tanzu/velero/pull/7845)
- solutions from call:
- requeue & mark as failed
- restart velero pod to trigger mark as failed
- set backup windows to avoid issue
### Discussion topics
- Daniel
- The DCO check issue:
- https://github.com/dcoapp/app/issues/211
- Shall we hold PRs for a few days?
- https://github.com/vmware-tanzu/velero/pull/7793
- Tiger - update docs
# May 28, 2024 (US/Europe)
### Attendees
- Scott Seago (Red Hat)
- Shubham Pampattiwar (Red Hat)
- Mateus Oliveira (Red Hat)
- Raghuram Devarakonda (CloudCasa)
- Tiger Kaovilai (Red Hat)
- Michal Pryc (Red Hat)
- Wes Hayutin (Red Hat)
### Status
- Scott
- Updated backup performance enhancement design PR, responded to comments
- https://github.com/vmware-tanzu/velero/pull/7628
- A few updates to the design forthcoming, in response to recent comments
- Shubham
- PR reviews and testing for velero 1.14
### Discussion topics
# May 22, 2024 (China/US)
### Attendees
- Yonghui Li (Broadcom)
- Xun Jiang (Broadcom)
- Shubham Pampattiwar (Red Hat)
- Jiaolin Yang (Broadcom)
- Tiger Kaovilai (Red Hat)
- Wes Hayutin (Red Hat)
- Yunqian Chen (Dell)
### Status
1.14.0: Preparing RC. Fixing problems with some corner cases for VolumePolicyExtension https://github.com/vmware-tanzu/velero/pull/7794
### Discussion topics
# May 14, 2024 (US/Europe)
### Attendees
- Shubham Pampattiwar (Red Hat)
- Michal Pryc (Red Hat)
- Raghuram Devarakonda (CloudCasa)
- Anshul Ahuja (Microsoft)
- Mateus Oliveira (Red Hat)
- Scott Seago (Red Hat)
- Tiger Kaovilai (Red Hat)
- Daniel Jiang (Broadcom)
- Sachin Singla (Red Hat)
- Wenkai Yin (Broadcom)
### Status
- Shubham
- Work on addressing feedback on Removal of snapshotVolumes flag necessity for volume policy snapshot action https://github.com/vmware-tanzu/velero/pull/7786
### Discussion topics
- Scott
- Backup performance enhancement design discussion
- https://github.com/vmware-tanzu/velero/pull/7628
- Responded to the last issue comment after last week's discussion. I'm not sure we need a lengthy discussion this week until the recent comment has been read through, but if there are specific questions today, we can go through them.
# May 8, 2024 (China/US)
### Attendees
- Yonghui Li (Broadcom)
- Xun Jiang (Broadcom)
- Ming Qiu (Broadcom)
- Scott Seago (Red Hat)
- Shubham Pampattiwar (Red Hat)
- Danfeng Liu (Broadcom)
- Tiger Kaovilai (Red Hat)
- Wenkai Yin(Broadcom)
- Shawn Hurley (Red Hat)
- Jiaolin Yang (Broadcom)
### Status
1.14.0: Testing and preparing for rc
- Yonghui
- Manual test and RC issue fixes
- Xun
- Manual test and bug fix.
- Ming
- Manual test and bug fix.
- Shubham
- Documentation PR for Volume Policy Extension: https://github.com/vmware-tanzu/velero/pull/7779
- Work on fixing (add more logs when a volume is skipped) [https://github.com/vmware-tanzu/velero/issues/7774]
### Discussion topics
- Daniel
- Discuss the date for rc.1 (late May)
- Scott
- Backup performance enhancement design discussion
- https://github.com/vmware-tanzu/velero/pull/7628
- Will respond point-by-point to the most recent comment later this week, but first we should probably have a discussion here, as some things may be getting lost in the back-and-forth comments.
# Apr 30, 2024 (US/Europe)
### Attendees
- Scott Seago (Red Hat)
- Raghuram Devarakonda (CloudCasa)
### Status
- Scott
- Updated backup performance enhancement design PR, responded to comments
- https://github.com/vmware-tanzu/velero/pull/7628#issuecomment-2083819288
- Link goes to last comment, summarizing current issues with a follow-on response to each.
- We should probably schdule a separate meeting to discuss at some point next week.
### Discussion topics
# Apr 24, 2024 (China/US)
### Attendees
- Yonghui Li (Broadcom)
- Xun Jaing (Broadcom)
- Scott Seago (Red Hat)
- Jiaolin Yang (Broadcom)
- Shubham Pampattiwar (Red Hat)
- Wes Hayutin (Red Hat)
- Tiger Kaovilai (Red Hat)
- Wenkai Yin(Broadcom)
- Danfeng Liu (Broadcom)
- Ming Qiu (Broadcom)
### Status
1.14.0: FC in this week, will launch post-FC manual testing
- Yonghui
- PR reviews and prepare test cases
- Xun
- Write manual test cases
- Daniel
- Prepare for FC
- Scott
- Updated backup performance enhancement design PR, responded to comments
- https://github.com/vmware-tanzu/velero/pull/7628
- See last few comments, thanks to Lyndon for the detailed response
- I think we agree on the use cases that need this design vs. those that don't.
- Regarding the remaining two points of disagreement listed, I think we can agree on the first point -- relaxing the v3-for-all-plugins requirement.
- On the second point, we still need the plugin API change to meet additional requirements (see the comments for details)
- Shubham
- Worked on Extend Volume Policies Impl PR: https://github.com/vmware-tanzu/velero/pull/7664
- Tiger
- Researching TLS certificates for [Design: Velero client download APIServer](https://github.com/vmware-tanzu/velero/pull/7344)
- Slack user support
### Discussion topics
- Xun
- [Data-only restore](https://hackmd.io/MIp_MAFXRoatsx0KANpDLA?view) discussion
- Daniel
- https://github.com/vmware-tanzu/velero/issues/7356
- Moving it out of v1.14?
- Discuss the requirement for restoring ownerReference. Does RH users have such requirement?