Velero Community Meeting and Open Discussion Notes

# Velero Community Meeting and Open Discussion Notes _This doc is meant to serve as the "one doc to rule them all" for Velero community meetings and open discussions._ _Join our [Google Group](https://groups.google.com/forum/#!forum/projectvelero) to get updates on the project and invites to community meetings._ _Community meetings are held every Tuesday at 12PM ET / 9AM PT ([Convert to your time zone](http://www.thetimezoneconverter.com/?t=09:00&tz=PT%20%28Pacific%20Time%29)) - [Zoom link](https://vmware.zoom.us/j/551441444)_ _You can find recordings of these meeting [on YouTube](https://www.youtube.com/playlist?list=PL7bmigfV0EqQRysvqvqOtRNk4L5S7uqwM)._ _[Archive of older meeting notes](https://hackmd.io/@jonasrosland/SyV22skpL)_ _ Add your request, question or suggestion to [our issue list](https://github.com/vmware-tanzu/velero/issues) _If you have anything you'd like to put on the agenda, please do so below for the next meeting:_ # Oct 27, 2020 ### Status Updates - carlisia - Suport-a-thon last week - Experimented with an internal tool to boot up an AWS EC2 cluster - Tested and documented - Neat (but dangerous) OSS tool to decommision AWS infrastructure: https://github.com/rebuy-de/aws-nuke - Sample config file: https://gist.github.com/carlisia/c687505ff9b89b429557f897659e1c49 - PR reviews + Helm - ashish - Support-a-thon last week - Community support this week - Have you heard about our [Velero Office hours](https://hackmd.io/I3u1x0u9T46KhuYZN4LX-A)? _this opens another hackmd page_ - dsmithuchida - Support-a-thon last week - Working on basic e2e test, mostly working - nrb - Support-a-thon last week - Revisting our bug triage process - Released v1.4.3 and v1.5.2 last week with a [security fix](https://github.com/vmware-tanzu/velero/security/advisories/GHSA-72xg-3mcq-52v4) - please upgrade when you can. - Blocker - [several design PRs](https://github.com/vmware-tanzu/velero/pulls?q=is%3Apr+is%3Aopen+label%3ADesign) are waiting on other maintainers to review ### Discussion Topics - Rafael @brito-rafa , Frankie @codegold79 and Jenting @jenting - Migration doc: https://github.com/vmware-tanzu/velero/issues/2551 . About K8s API Groups: http://www.rafaelbrito.com/2020/04/kubernetes-api-groups.html - Alay: Provide more visibility on reasons for PartialFailure by reporting the errors on velero CR - The CRs don't provide errors directly, just he number - CRs are limited to 1.5mb usually due to etcd - There's a mismatch between restore & backup log formats, making just parsing difficult - This could be a subresource backed by a web hook, not a CRD ### Community Shoutouts - [@a-mccarthy](https://github.com/a-mccarthy): [fix minio code samples](https://github.com/vmware-tanzu/velero/pull/3034) - [@cvhariharan](https://github.com/cvhariharan): [Add warning to velero version cmd.](https://github.com/vmware-tanzu/velero/pull/3024) - [@doughepi](https://github.com/doughepi): [Adding fix for restic init container index on restores.](https://github.com/vmware-tanzu/velero/pull/3011) - [@jenting](https://github.com/jenting): [[velero] Restrict release on upstream repo](https://github.com/vmware-tanzu/helm-charts/pull/174) - [@mukulgit123](https://github.com/mukulgit123): [[velero] Correct syntax of iam role pod annotation in values.yaml](https://github.com/vmware-tanzu/helm-charts/pull/170) - [@machine424](https://github.com/machine424): [[velero] Add priorityClassName in the values.yaml file to make it 'official'](https://github.com/vmware-tanzu/helm-charts/pull/159) # Oct 20, 2020 ### Status Updates - carlisia - Community support status: - done: triaged all recent (couple weeks) issues - wip: triaging older issues and starting on the other repos - PR reviews - tackle my broken test for https://github.com/vmware-tanzu/velero/pull/3004 (DownloadRequest migration to kubebuilder) - nrb - Fixed https://github.com/vmware-tanzu/velero/issues/2570, plan to get v1.5.2 & v1.4.3 out this week. - Planning our support-a-thon to reduce the number of open support issues this week, Wednesday to Friday - v1.6 checkup! - bridget - Made some changes to [#2993](https://github.com/vmware-tanzu/velero/pull/2993) based on comments - Discovered a problem with approach in newest changes so need to revisit that approach and modify tests - Helped Ashish with internal release processes last week - Reviewing design docs - dsmithuchida - PTO last week - Working on e2e tests ### Discussion Topics - carlisia - Should we add this to our list of supported plugins: https://github.com/Lirt/velero-plugin-swift/ - How do we handle requests for a new plugin: https://github.com/vmware-tanzu/velero/issues/2988 - Add them to https://github.com/vmware-tanzu/velero/tree/main/site/content/plugins/list - Also https://github.com/vmware-tanzu/velero/blob/main/site/content/docs/main/supported-providers.md - We are hiring a senior Product Manager to lead Velero: https://twitter.com/carlisia/status/1318304806784573440 (spread the word please!) - Jonas got a promotion, congratulations!! Well deserved title of Head of Community Management - nrb - What do folks think about moving from ZenHub to GitHub Projects? - Example: https://github.com/orgs/projectcontour/projects/2 - Advantages: - Integrated (doesn't need a 3rd party site/plugin) - Can create multiple project boards for different things - Disadvantages: - GitHub issues aren't automatically added to the board (maybe GitHub Actions exist for this?) - https://github.com/marketplace/actions/github-project-automation - No concept of epics - One time switch cost - v1.6 priorities - Implementing e2e testing - [Continue migrating to kubebuilder/controller-runtime](https://github.com/vmware-tanzu/velero/issues/2597) - [Enable the use of custom credential providers](https://github.com/vmware-tanzu/velero/pull/2403) - Upgrade to restic v0.10.0 - Phuong - Backup in parallel https://github.com/vmware-tanzu/velero/issues/2888: maybe just backing up pods in parallel - Enhance Backup with plugin timeout (https://github.com/vmware-tanzu/velero/issues/2922): pull request submitted https://github.com/vmware-tanzu/velero/pull/3018 - dymurray - Follow up from previous meeting, restore progress design proposal: https://github.com/vmware-tanzu/velero/pull/3016 ### Community Shoutouts - [@betta1](https://github.com/betta1): [Fix BSL controller to avoid invoking init() on all BSLs regardless of…](https://github.com/vmware-tanzu/velero/pull/2992) - [@gliptak](https://github.com/gliptak): [Bump Go to 1.15](https://github.com/vmware-tanzu/velero/pull/2974) - Arianit Uka for reporting https://github.com/vmware-tanzu/velero/issues/2570 - Pranav Gaikwad for https://github.com/vmware-tanzu/velero/pull/3016 # Oct 13, 2020 ### Status Updates - carlisia - DownloadRequest migration to kubebuilder/runtime-controller: https://github.com/vmware-tanzu/velero/pull/3004 - Testing https://github.com/vmware-tanzu/velero/issues/2991 (BSL controller loop always invokes init() regardless of backup location's ValidationFrequency) - PR reviews - nrb - Code for https://github.com/vmware-tanzu/velero/issues/2570, but need to test against clusters now. - Working on community issue support process with Carlisia to help improve response time on GitHub issues. - Need to get the Azure and CSI plugin updated and out in the next day or so. - Noticed PRs that go from draft to open aren't getting processed by the review Action; may look into it, or just file an issue. - bridget - PR for deleting backups when cloud resources have been deleted (https://github.com/vmware-tanzu/velero/pull/2993). Need to follow up on comments from @nrb. ### Discussion Topics - Phuong - enhance BackupItemActionPlugin with timeout https://github.com/vmware-tanzu/velero/issues/2922 - Aspects to this are in the [wait for AdditionalItems to be ready](https://github.com/vmware-tanzu/velero/pull/2867) - [nrb] Can we make timeouts work for both? - [nrb] Are the timeouts per plugin, per plugin type, or universal? - sseago - https://github.com/vmware-tanzu/velero/pull/2867 was for a specific CR in OpenShift - [nrb] there's a use case for the TAS Postgres operator now too - dymurray - Supporting restore progress reporting https://github.com/vmware-tanzu/velero/issues/21 - Tackle restores similar to the backups right now as a first phase - Current issue is on restore we don't have a fully accurate number of items. For example, when using `--include-namespaces`, we don't know the number of to-be-restored resources ahead of time. - Getting a 2 pass restore system would help immensely here, since we would pre-process and match resources before sending them to the API server. This would give us an overall count to report for progress. - Antony - https://github.com/vmware-tanzu/velero/issues/2991: - BSL controller initializes backup storage locations almost every second - Significant issue for object stores where init has expensive calls (e.g API calls) - Any chance we could get a hotfix? When does v1.5.2 ship? ### Contributor Shoutouts - [@mickkael](https://github.com/mickkael): [Allow Timezone change in the container](https://github.com/vmware-tanzu/velero/pull/2944) - [@jenting](https://github.com/jenting): [[velero] Add helm annotation for velero.io/v1 schedule CRD](https://github.com/vmware-tanzu/helm-charts/pull/153) # Oct 6, 2020 ### Status Updates - dsmithuchida - Working on e2e testing and ginkgo framework - bridget - Couple of outstanding PRs that I'd like reviewed ([#2951](https://github.com/vmware-tanzu/velero/pull/2951), [#2969](https://github.com/vmware-tanzu/velero/pull/2969)) - Been working on some bugs: - [Backup deletion fails if cloud resources already deleted #2980](https://github.com/vmware-tanzu/velero/issues/2980) This is a regression in behaviour from v1.4.2 due to the introduction of DeleteItemActions. We can introduce a fix, but how should we handle the case where there are DeleteItemActions to perform but the resources are deleted? - [Unable to install Velero using go get #2966](https://github.com/vmware-tanzu/velero/issues/2966) We can fix this by upgrading the version of CSI external-snapshotter but it requires updating the version of Kubernetes modules we're using to 0.19.0. What is the impact of upgrading these libraries? - nrb - Not a whole lot of updates; was on semi-PTO last week. - Intending to do a "support-a-thon"/bug smash session when the whole team is back ### Discussion Topics - [Waiting for Additional Items to restore](https://github.com/vmware-tanzu/velero/pull/2867) - Issue #2980 - deleteitemactions regression - Happens all the time, but what do we do when there's DeleteItemActions but no backup tarball? - Since there's no way to really know what to match the plugins against, log that there was no tarball and continue, skipping the plugins. ### Community Contributions - [@sseago](https://github.com/sseago): [restore proper lowercase/plural CRD resource](https://github.com/vmware-tanzu/velero/pull/2949) - [@svcua](https://github.com/svcua): [[velero] Bump Velero version to v1.5.1](https://github.com/vmware-tanzu/helm-charts/pull/163) - [@xUnholy](https://github.com/xUnholy): [[velero] fix bug using workload identity issue #161](https://github.com/vmware-tanzu/helm-charts/pull/162) - [@nirav-chotai](https://github.com/nirav-chotai): [[Velero] Fixing restic daemonset yaml](https://github.com/vmware-tanzu/helm-charts/pull/155) # Sept 22, 2020 ### Status Updates - nrb - [Documenting plugin release process](https://github.com/vmware-tanzu/velero/pull/2952) - Need to get Azure and CSI plugins released - [Stop trying to push docker images on forks](https://github.com/vmware-tanzu/velero/pull/2953) - Reviews on main repo. - Thinking on ways to make Velero more concurrent/faster as part of v1.6; no concrete suggestions yet, I owe a high level goals doc per discussion last week. - Maintainers please take a look at https://github.com/vmware-tanzu/velero-plugin-for-csi/pull/70; needed for the next version of the CSI plugin - carlisia - Last week worked on v1.5.1 release - Reviews - Piling up: more reviews, GH issues + Helm chart - working on the CLI side of the download request migration to kubebuilder/controller-runtime. - bridget - Worked on some improvements to our release scripts following the 1.5.1 release - Now looking at internal build processes - dsmithuchida - Resource blocking in vSphere plug-in for Project Pacific internal resources - OpenSource project approvals for GVDDK (currently part of Astrolabe) and Data Generator (Kibishii) test tool ### Discussion Topics - nrb: [Defining prerelease to GA timeframes](https://github.com/vmware-tanzu/velero/issues/2935) - From last week: - Prereleases ~1 week between each? - RC -> GA, ~2 weeks? - Phuong - 3 month release cycle is acceptable for them, waiting for features on that timeframe is reasonable. Longer is too much. - The RC was helpful to integrate and test with their product. It meant updating to the actual release was just removing some characters. - 1 week seemed reasonable to them, but if they hit bugs it may not be enough time. - Dylan - Red Hat's Konveyor lags a little. They want to support older Kubernetes releases like v1.7 due to supporting OpenShift 3. (this isn't determined for upstream Velero) - Konveyor does OpenShift 3->4 and 4->4 migration. - Red Hat also has OADP for OpenShift data protection, and their Velero fork is used there, too. - Red Hat likes the 3 month release cycle, especially for OADP. Backwards compatibility in Konveyor is trickier. - For OADP, they tried the RC for basic tasks. - poojita: Recover OpenShift's native resource: DeploymentConfig - Velero restore error seen has been captured below: - Velero failed to restore namespace frank3. {\"namespaces\":{\"frank3\":[\"error restoring imagetags.image.openshift.io/frank3/httpd-example:latest: ImageTag.image.openshift.io \\\"httpd-example:latest\\\" is invalid: spec: Required value: spec is a required field during creation\"]}}. Velero restore '51685651-5369-51e1-88df-8977874919ca-2020-09-16-08-53-48-frank3' failed: {\n \"phase\": \"PartiallyFailed\",\n \"warnings\": 6,\n \"errors\": 1\n}." - Dylan - There's an OpenShift plugin that can help restore these CRDs on vanilla Velero installs. https://github.com/konveyor/openshift-velero-plugin - For ImageTag, it skips restore of these - Plugin recreates ImageStream, which then recreates the ImageTag. - ImageTag is a new, undocumented resource in OpenShift v4.4. - The plugin is used in the context of the migration product (Konveyor) and data protection ([OADP](https://github.com/konveyor/oadp-operator)). - Can file GitHub issues on that repo if you have issues/questions. - OADP bundles Velero and the AWS plugin on OpenShift. The benefit here is that it backs up the images to the S3 bucket. - Plugin can't do it by itself right now. OADP sets up a image repo that the plugin doesn't orchestrate right now - Alay - Plugin dependencies are challenging - If you deploy Velero by itself, without a wrapper, there are challenges in connecting plugins to external systems such as StatefulSets or Deployments. - Dave: Difference between the vSphere plugin and OADP is that OADP is at what point in the lifecycle they're active. - May be able to extend plugins via the Velero app operator that VMware is working on. - RH's requirements: - Before the plugin runs, ensure the dependency is healthy - During/after backup, ensure the dependency is healthy. - If it's not, short circuit the operation instead of trying the full backup operation and failing. - Narashima - Took a backup w/ Velero and tried to restore it to another cluster. ReplicaSets/Deployments are getting duplicated in the new cluster. - kubectl get shows multiple entries. ### Shoutouts Slow week for contributions, but thanks to mikkael for the PR to [allow users to change the container's timezone](https://github.com/vmware-tanzu/velero/pull/2944) # Sept 15, 2020 ### Status Updates - ashish - [v1.5.0-rc.1](https://github.com/vmware-tanzu/velero/releases/tag/v1.5.0-rc.1) released on 9/10 - Getting exec restore hooks merged. 🎉 Thank you Andrew Reed! - Testing the RC release. - Helping out with hardening the release procedure. - Getting PRs merged and fixing some documentation bugs. - Migration to kubebuilder: - BackpController- work started and made some progress. - nrb - Was on PTO last week, not a whole lot to discuss there - Would like to get v1.5.0 out for sure this week - Following up on some PR reviews that were opened while I was still out - Looking at design docs for the v1.6 timeframe - [Multiple secrets per plugin](https://github.com/vmware-tanzu/velero/pull/2403) - Concurrent backup/restore job support - Welcome Dave to the team officially! - carlisia - Did the RC release last week (Ashish collaborated) - Reorganized the release instructions: https://github.com/vmware-tanzu/velero/pull/2931 - Have the download request migration on my list for this week - bridget - Still getting up to speed with things - Want to get [#2881](https://github.com/vmware-tanzu/velero/pull/2881) reviewed/merged - Looking for something new to start on - stephanie - 1.6 stack rank design and implementation - some delays getting this published and we have a deadline for getting that up and published for community viewing this week - we encourage comments and feedback to the stack rank - any roadmap updates will also be published following the finalized 1.6 stack rank - dave - Vsphere plugin testing on 1.5 rc looking good - E2E testing: - something developers can run - Nolan: expect this to be open source - something that goes into CI - ### Discussion Topics - nrb: Concurrent backup/restore job support - This is at the _job_ level, not within a job (for example, grabbing all PVs in parallel) - There was a proposal for implementing this as worker pods, but I'd like to look at different design approaches, not just one - Would like to start this design work, but it's going to take a while. Will initially get a design doc with goals/non-goals up. - Things to address: - quiescing at backup - contention at restore time - Locking - Become more distributed - [Carlisia] Sounds more like product requirements - Can we implement concurrency piecemeal? - For example, is it acceptable to be concurrent only on a namespace level? - Can we make PoC(s)? - **Action item** - get a doc with goals/non-goals that set up guard rails, then have a sessions to discuss it w/ whiteboarding - carlisia: - We now have a maintainer cheat sheet: https://velero.io/docs/main/maintainers/ - ashish: - What is the general timeline for a release: - Pre -> RC -> GA - Nolan: RC -> GA: ~2 weeks - Nolan: Release cadence is up to debate, though - Ashish: ~1 week between pre-releases. - dave: - Passing feature flags to init containers - The init containers just copy the plugin to a well-known location. - vSphere plugin has a bunch of ancillary stuff and we install from the init container - The `--features` flag is passed from the core velero to the plugin. I spoke with someone on the vpshere plugin to suggest this. ### Contributor Shoutouts - [@jenting](https://github.com/jenting): [Show format version on velero backup describe](https://github.com/vmware-tanzu/velero/pull/2901) - [@areed](https://github.com/areed): [Restore exec hooks doc](https://github.com/vmware-tanzu/velero/pull/2896) # Sept 8, 2020 ### Status Updates - carlisia - RC this week - On community support - Nolan is on PTO - Working on the download request migration - ashish - 1.5 release testing - restore hooks PR: - testing - [PR 2804](https://github.com/vmware-tanzu/velero/pull/2804) reviews - getting that merged hopefully today. - bridget - Reviewing PRs for the 1.5 release ### Discussion Topics - Phuong: Timeout for plugins. For app consistent backup, the pod will be quiesced while snapshot being taken. If the PVC plugin taking the snapshot and that plugin take a long time (say because it fails and retries), the application will be blocked for a long time which is unwanted behavior. ### Contributor Shoutouts - [@jenting](https://github.com/jenting): [fix EnableAPIGroupersions output log format](https://github.com/vmware-tanzu/velero/pull/2882) - [@pawanpraka1](https://github.com/pawanpraka1): [fix: rename the PV if VolumeSnapshotter has modified the PV name](https://github.com/vmware-tanzu/velero/pull/2835) - [@arminc](https://github.com/arminc): [[velero] Adding imagePullSecrets to the chart](https://github.com/vmware-tanzu/helm-charts/pull/149) # Sept 1, 2020 ### Status Updates - carlisia - pto last Friday - addressing code reviews - going to review/test PRs for 1.5 - nrb - Planning for Q4 w/ VMware - Some planning for v1.6 - reviewing PRs - v1.5.0 delayed (in discussion topics) - Welcoming Dave! - Community support this week - On vacation next week, will transfer responsibilities over to folks prior - bridget - Finally transitioned over to the Velero project - Getting up to speed - Reviewing PRs for 1.5 ### Discussion Topics - nrb: v1.5.0 has been delayed. [We've got a list of release blocker PRs](https://github.com/vmware-tanzu/velero/pulls?q=is%3Apr+is%3Aopen+label%3Arelease-blocker) that we'd like to get through, and will not add more code changes unless there's a critical bug/security fix. The PRs are in relative priority. - Timeframe - ~2 weeks - Testing will be more automated starting in v1.6, but we don't have a defined e2e test suite right now, just whatever is in `make ci` - nrb: VMware will be having a company holiday on Friday, September 4 and Monday, September 7 will be US Labor Day. - carlisia: do we want to halt the Velero server in this case: "Invalid value for --disable-controllers flag provided" (https://github.com/vmware-tanzu/velero/blob/976484553062262324cc1304607ea00841262db8/pkg/cmd/server/server.go#L822) - **Action Item**: Carlisia to open an issue to discuss this and document the issue - phuong: - quick update on the backup cluster resources - Able to back up clustorerole and pv in the context of a namespace, don't need to modify the selection logic - backing up PVC in parallel - Taking backups of pods in logical group in a serial manner could lead to drift - This is parallelization w/in a backup job, not parallelization of all jobs - Currently this would be VolumeSnapshotter plugins - may not be a resource contention issue - This is the main issue here - use case is for Cassandra - How would this work with the CSI plugin? Since it's BackupItemAction, the interface is way different - At the Kubernetes API server level, would we want to specify parallelism per resource type? - CRDs may need this, too - Need to be careful to not overflow resource limits - Don't want to overwhelm writing to the backup tarball - Once we get the body, that could be funneled to the tarball serially - **Action Item**: Phuong to make an issue to continue discussion: https://github.com/vmware-tanzu/velero/issues/2888 - sseago: posted design doc for RestoreItemAction "wait for additional item to be ready" (https://github.com/vmware-tanzu/velero/pull/2867) - To add timeouts, we could use https://velero.io/docs/v1.4/custom-plugins/#plugin-configuration - brito-rafa: will work with jenting on supporting multiple API Groups (https://github.com/vmware-tanzu/velero/issues/2551) ### Contributor Shoutouts - [@a-mccarthy](https://github.com/a-mccarthy): [point contributors to style guide](https://github.com/vmware-tanzu/velero/pull/2872) - [@a-mccarthy](https://github.com/a-mccarthy): [add hugo default TOC](https://github.com/vmware-tanzu/velero/pull/2866) - [@a-mccarthy](https://github.com/a-mccarthy): [add new table shortcode](https://github.com/vmware-tanzu/velero/pull/2865) - [@a-mccarthy](https://github.com/a-mccarthy): [update docs to match style guide](https://github.com/vmware-tanzu/velero/pull/2861) - [@a-mccarthy](https://github.com/a-mccarthy): [Update release checklist to include more info around blog posts and r…](https://github.com/vmware-tanzu/velero/pull/2837) - [@c3y1huang](https://github.com/c3y1huang): [[velero] Fix helm 3.3.0 lint issue](https://github.com/vmware-tanzu/helm-charts/pull/150) # August 25, 2020 ### Status Updates - stephanie - building the priority/stack rank list for the 1.6 release - this will be published after we complete planning next week for the community to refer to and will include additional details that compliment our current product roadmap - ashish - Community support this week. - Misc work for the 1.5 release: - [Pre-release](https://github.com/vmware-tanzu/velero/releases/tag/v1.5.0-beta.1) is out. Try it out in your non-prod environments - Bug fix for server version not being correctly displayed as output of `velero version`. - Updating some docs - Getting PRs reviewed and merged. - Testing out the release - Will be taking another look at the exec restore hooks PR. - carlisia - community support last week - ServerStatusRequest PR (migration to kubebuilder): https://github.com/vmware-tanzu/velero/pull/2838 - wip: DownloadRequest migration to kubebuilder - nrb - released v1.5.0-beta.1 - testing the beta - please try it out in your environments, too! - planning for v1.6 - Owe a review on the restore hooks exec work ### Discussion Topics - nrb: We aim to have v1.5.0 out by September 1 - [Let us know what Velero code you depend on](https://github.com/vmware-tanzu/velero/issues/2654) - Defining printer columns - Do we need to get the exact same values as Velero's CLI in kubectl printer columns? - Not necessarily - they can be raw values, but we can use webhooks for calculations. - Phuong updating go modules - 1.5.0-beta.1 generating CRs having issues - The error has an issue with missing a context argument - related to upgrading to Kubernetes v1.18 ``` ERROR] generated/clientset/versioned/typed/applicationtemplate/v1beta1/applicationtemplate.go:73:5: not enough arguments in call to c.client.Get().Namespace(c.ns).Resource("applicationtemplates").Name(name).VersionedParams(&options, scheme.ParameterCodec).Do [ERROR] have () [ERROR] want (context.Context) ``` ### Contributor Shoutouts - [@a-mccarthy](https://github.com/a-mccarthy): [add index files to api types folder](https://github.com/vmware-tanzu/velero/pull/2839) - [@ipochi](https://github.com/ipochi): [docs: add metadata to resource-filtering.md](https://github.com/vmware-tanzu/velero/pull/2832) - [@bgagnon](https://github.com/bgagnon): [Override logrus.ErrorKey when json logging is enabled](https://github.com/vmware-tanzu/velero/pull/2830) - [@jenting](https://github.com/jenting): [doc: fix feature flag EnableCSI on velero backup describe command only](https://github.com/vmware-tanzu/velero/pull/2817) # August 18, 2020 ### Status Updates - ashish - Mostly preparing for 1.5 release and addressing any outstanding items, doc fixes and PR reviews. - I owe @areed another round of review on the exec restore hooks PR. Going to do that today. - During investigation of [Issue #2789](https://github.com/vmware-tanzu/velero/issues/2789) added some [documentation](https://github.com/vmware-tanzu/velero/wiki/Contributing-FAQ#how-do-i-use-minio-with-https-using-a-custom-ca-as-an-object-store-for-velero) to our wiki on setting up standalone MinIO object storage using custom ca cert and self-signed certificates with https to use with Velero - nrb - Worked on adding the DeleteItemActions plugin type in https://github.com/vmware-tanzu/velero/pull/2808 and https://github.com/vmware-tanzu/velero/pull/2815 - Reviews - Looking to review release process w/ Abbie in light of the v1.5 beta release - carlisia - On community support this week - Lots of PR reviews last week - wip (ServerStatusRequest -> kubebuilder): https://github.com/vmware-tanzu/velero/compare/main...carlisia:c-status-request ### Discussion Topics - Phuong Hoang: avoid backup PersistentVolumes not referenced by included namespaces when enable --include-cluster-resources - Use --exclude-resources=persistentvolumes? - We still want to include the PersistentVolumes being used by PVCs belong to the namespace being backed up but not including other PersistentVolumes - Plugin additional items _may_ ignore this, but we can't remember - Also want to include other cluster-scoped resources like ClusterRoleBindings - Sounds like we want OR selector logic support - One idea - use --include-resources and specify exactly what you want - Limitation: you have to specify _exactly_ what you want - **Action item**: Phuong will investigate a proposal to enhance item selection. - stephanie - Velero 1.6 release planning (starting around v1.5 beta release) - We'll share this around Sept 1 in ZenHub & GitHub - Will update roadmap.md, too - Between now and then, please :+1: any issues that you're interested in seeing in the v1.6 release - [Let us know what Velero code you depend on](https://github.com/vmware-tanzu/velero/issues/2654) - Ashish: Velero Office Hours are coming, exact dates & times to be announced ### Contributor Shoutouts - [@a-mccarthy](https://github.com/a-mccarthy): [add note about windows support](https://github.com/vmware-tanzu/velero/pull/2806) - [@runzexia](https://github.com/runzexia): [add kindfor func to get apiresource from gvk](https://github.com/vmware-tanzu/velero/pull/2764) - [@jenting](https://github.com/jenting): [[velero] bump velero version to v1.4.2](https://github.com/vmware-tanzu/helm-charts/pull/145) - [@jenting](https://github.com/jenting): [fix: pull request template not working](https://github.com/vmware-tanzu/helm-charts/pull/144) - [@cpanato](https://github.com/cpanato): [Update GH-pages branch](https://github.com/vmware-tanzu/helm-charts/pull/141) - [@cpanato](https://github.com/cpanato): [actions: update helm actions to use release 1.0.0](https://github.com/vmware-tanzu/helm-charts/pull/140) - [@jenting](https://github.com/jenting): [Add Github issue template](https://github.com/vmware-tanzu/helm-charts/pull/137) - [@jenting](https://github.com/jenting): [fix: custom CA not working](https://github.com/vmware-tanzu/helm-charts/pull/135) # August 11, 2020 ### Status Updates - carlisia - PTO last week - Reviewing PRs - Back to the last of migrating ServerStatusRequest to kubebuilder - ashish - Community support last week - Got to a bunch of PRs and GH issues. - Restore Hooks: - PR for Init container restore hooks [PR 2787](https://github.com/vmware-tanzu/velero/pull/2787) - PR to fix API tyes [PR 2793](https://github.com/vmware-tanzu/velero/pull/2793) - Fix to pass the `--default-volumes-to-restic` flag from schedules to the backup options. [PR 2776](https://github.com/vmware-tanzu/velero/pull/2776) - Setup CRD validation CI in GitHub actions [PR 2805](https://github.com/vmware-tanzu/velero/pull/2805) - Currently setup to run against kubernetes version 1.16.9, 1.17.0 and 1.18.4 - AI: Switch to latest of each minor release - Restructuring build broke the EntryPoint for the restic-restore helper. [PR 2802](https://github.com/vmware-tanzu/velero/pull/2802) - PRs in the CSI plugin repo: - Updating the CI in the repo to push images with `latest` tag. [PR 65](https://github.com/vmware-tanzu/velero-plugin-for-csi/pull/65) - Better handling of restore when VolumeSnapshot resources exist but the associated workloads got deleted [PR 64](https://github.com/vmware-tanzu/velero-plugin-for-csi/pull/64) - nrb - Community support this week - Working on DeleteItemAction [implementation](https://github.com/vmware-tanzu/velero/pull/2808) - [Design doc](https://github.com/vmware-tanzu/velero/pull/2713) - Trying to get GitHub Actions working to automatically assign reviewers on PRs when opened. (https://github.com/vmware-tanzu/velero/pull/2807) - Aiming to have a 1.5 beta out by August 21st at the latest. - stephanie - We are aiming to have 1.6 release planning start following the 1.5 beta at the end of August - refer to our roadmap for items currently planned for December 2020 which corresponds with the [1.6 release](https://github.com/vmware-tanzu/velero/pull/2548/files?short_path=38574c0#diff-38574c080d4e2eb38c49b86e6588ad98) - [Velero security policy published](https://github.com/vmware-tanzu/velero/pull/2797/files?short_path=9005539#diff-9005539790e22f749a81424d29a6777f) ### Discussion Topics - Introducing Bridget! - [Let us know what Velero code you depend on](https://github.com/vmware-tanzu/velero/issues/2654) - Velero-plugin-for-CSI [PR 1902](https://github.com/kubernetes/enhancements/pull/1902) KEP to add validation hooks on the VolumeSnapshot API objects. - Took a look at it and these don't seem to violate our assumptions/expectations - Would love to have more 👀 on it - What versions of Kubernetes does Velero support? - Alay: Red Hat supports all the way back to Kube v1.7, will bump up to v1.9. Uses Velero v1.3 - An LTS ### Contributor Shoutouts - [@alexcpn](https://github.com/alexcpn): [Update minio.md](https://github.com/vmware-tanzu/velero/pull/2799) - [@bgagnon](https://github.com/bgagnon): [Always use groupResource.String() when logging (fixes #2795)](https://github.com/vmware-tanzu/velero/pull/2796) - [@doughepi](https://github.com/doughepi): [Add the ability to set the allowPrivilegeEscalation property on the Restic restore helper via plugin ConfigMap](https://github.com/vmware-tanzu/velero/pull/2792) - [@RobReus](https://github.com/RobReus): [Reverting change on 1.4 docs and re-applying to main docs](https://github.com/vmware-tanzu/velero/pull/2791) - [@skriss](https://github.com/skriss): [update CI badge on README](https://github.com/vmware-tanzu/velero/pull/2786) - [@a-mccarthy](https://github.com/a-mccarthy): [update docs to include cpu/memory defaults for restic](https://github.com/vmware-tanzu/velero/pull/2772) - [@mtritabaugh](https://github.com/mtritabaugh): [Add resource filtering page](https://github.com/vmware-tanzu/velero/pull/2771) - [@RobReus](https://github.com/RobReus): [Refactor image builds to use buildx for multi arch image building](https://github.com/vmware-tanzu/velero/pull/2754) - [@cfreemoser](https://github.com/cfreemoser): [Fixed wrong environment variable in Readme](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/pull/60) - [@sabvente](https://github.com/sabvente): [Remove extra indentation from README](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/pull/59) # August 4, 2020 ### Status Updates - ashish - Community support this week - Have a list of PRs to review - New issues that need triage - Working on restore hooks - [PR 2787](https://github.com/vmware-tanzu/velero/pull/2787) to implement the init container restore hooks. - Need to run this through tests before removing WIP. - nrb - Out sick late last week - Working on [DeleteItemAction design](https://github.com/vmware-tanzu/velero/pull/2713). Please take a look! - Investigating options on project management and release cadence to get issues handled more quickly, as well as get fixes out more regularly. - carlisia - PTO! - Working on migrating ServerStatusRequest to kubebuilder ### Discussion Topics - [Let us know what Velero code you depend on](https://github.com/vmware-tanzu/velero/issues/2654) - Andrew Reed - need any help with exec restore hook? - How do we mitigate people being on PTO? Slackening the reviewer constaints? - Storj persistent volume support - **Goal**: back up any Kubernetes PV to Storj object store - rclone/restic support? - treating volume as a block device? - vSphere plugin currently has a data mover - https://github.com/vmware-tanzu/velero-plugin-for-vsphere - github.com/openebs/velero-plugin has a data mover too - Let's start a thread on the [Google Group](https://groups.google.com/g/projectvelero) ### Contributor Shoutouts - [@jenting](https://github.com/jenting): [Add a flag cacert for velero backup-location create](https://github.com/vmware-tanzu/velero/pull/2778) - [@jenting](https://github.com/jenting): [fix: helm upgrade did not work correctly on BSL and VSL configuration](https://github.com/vmware-tanzu/helm-charts/pull/133) - [@MrFreezeex](https://github.com/MrFreezeex): [Fix Helm 2 CRD generation](https://github.com/vmware-tanzu/helm-charts/pull/126) - [@alikhtag](https://github.com/alikhtag): [[impovement] Add podLabels to the helm chart](https://github.com/vmware-tanzu/helm-charts/pull/120) - [@kyounger](https://github.com/kyounger): [fix: Restic daemonset SA always set, like velero deployment](https://github.com/vmware-tanzu/helm-charts/pull/119) - [@fredgate](https://github.com/fredgate): [Add log configuration to Restic](https://github.com/vmware-tanzu/helm-charts/pull/117) - [@cesarokuti](https://github.com/cesarokuti): [add schedule labels](https://github.com/vmware-tanzu/helm-charts/pull/115) - [@ckotzbauer](https://github.com/ckotzbauer): [add affinity option to velero deployment](https://github.com/vmware-tanzu/helm-charts/pull/107) # July 28, 2020 ### Status Updates - carlisia - Community support this week - PTO next week - Bringing the Helm repo up to date with v1.4 + more Velero Helm TLC - Continue work on the ServerStatusRequest migration to kubebuilder - ashish - Restore hooks: - Adding constants [PR#2750](https://github.com/vmware-tanzu/velero/pull/2750) - Updating RestoreSpec [PR#2761](https://github.com/vmware-tanzu/velero/pull/2761) - Backing up pod volumes using restic by default: - [PR 2611](https://github.com/vmware-tanzu/velero/pull/2611) - Excluding volumes mounting secrets and config maps - Opening Issue [2770](https://github.com/vmware-tanzu/velero/issues/2770). Support for label selector to choose pods to which the `defaultVolumesToRestic` should be applied. :thumbsdown: / :thumbsup: in the issue - PR reviews - nrb - Working on some CI improvements around assignment/reviewers, but ran into issues with how GitHub Actions expose secrets - PR reviews - Intending to get DeleteItemAction design doc updated today or tomorrow - Was on community support last week ### Discussion Topics - Welcome JenTing Hsiao (https://github.com/jenting) as an official maintainer of the Velero Helm chart! - Let us know what parts of the Velero API you use - https://github.com/vmware-tanzu/velero/issues/2654 ### Contributor Shoutouts - [@areed](https://github.com/areed): [Add hooks fields to restore context](https://github.com/vmware-tanzu/velero/pull/2755) - [@thejasbabu](https://github.com/thejasbabu): [Add Start and Completion Timestamp to Restore Status](https://github.com/vmware-tanzu/velero/pull/2748) #### Helm Chart - [@jenting](https://github.com/jenting): [fix: helm upgrade did not work correctly on BSL and VSL configuration](https://github.com/vmware-tanzu/helm-charts/pull/133) - [@jenting](https://github.com/jenting): [helm 2,3 CI test + Upgrade CRDs to v1.4](https://github.com/vmware-tanzu/helm-charts/pull/125) - [@MrFreezeex](https://github.com/MrFreezeex): [Fix Helm 2 CRD generation](https://github.com/vmware-tanzu/helm-charts/pull/126) - [@cesarokuti](https://github.com/cesarokuti): [add schedule labels](https://github.com/vmware-tanzu/helm-charts/pull/115) - [@tomahawk28](https://github.com/tomahawk28): [Helm 2,3 supports CRD at the same time](https://github.com/vmware-tanzu/helm-charts/pull/109) - [@ckotzbauer](https://github.com/ckotzbauer): [add affinity option to velero deployment](https://github.com/vmware-tanzu/helm-charts/pull/107) # July 21, 2020 ### Status Updates - ashish - Getting Release 1.4.2 released and testing the release. - PR reviews and issue investigation - PR to add metrics from restic backup [2719](https://github.com/vmware-tanzu/velero/pull/2719) - nrb - PR reviews - [Breakdown of restore hooks work](https://app.zenhub.com/workspaces/velero-5c59c15e39d47b774b5864e3/board?labels=restore%20hooks&milestones=v1.5%232020-08-31&repos=99143276,112385197,213946861,190224441,214524700,214524630) - Merged insensitive language PRs - Added design doc for [DeleteAction plugins](https://github.com/vmware-tanzu/velero/pull/2713) - I have a couple small PRs ready for review: https://github.com/vmware-tanzu/velero/pull/2735 & https://github.com/vmware-tanzu/velero/pull/2705 - carlisia - Community support last week - We now have a BSL controller - Wrapping up conversion of serverStatusRequest to use controller-runtime - Fixing a bug on the verify script - We started filling out our FAQ page: https://github.com/vmware-tanzu/velero/wiki/FAQ - stephanie - restore hooks product requirements reviews (https://github.com/vmware-tanzu/velero/pull/2699) ### Discussion Topics - - stephanie - discussion restore hooks use case #7 data quiescing (https://github.com/vmware-tanzu/velero/pull/2699/files#diff-1e4163f56a0d34eb56295c1901279753) - dave - DeleteAction discussion - Started as a high level delete, but wanted a DeleteItemAction - Act on items in the backup tarball. - Options: - Pull down the whole tarball and look through it - Add index of things that BackupItemActions created - Reminder to let us know if you're using any Velero code at https://github.com/vmware-tanzu/velero/issues/2654 ### Community Shoutouts - Phuong for testing out the main branch and reminding us we need to get update instructions documented - [@kathpeony](https://github.com/kathpeony): [Add backupValidationFailureTotal to metrics](https://github.com/vmware-tanzu/velero/pull/2714) - [@fvsqr](https://github.com/fvsqr): [StorageGrid compatibility ](https://github.com/vmware-tanzu/velero/pull/2712) - [@laverya](https://github.com/laverya): [k8s 1.18 import](https://github.com/vmware-tanzu/velero/pull/2651) - [@bryanro92](https://github.com/bryanro92): [[FIX] add --bucket param to workload identity install](https://github.com/vmware-tanzu/velero-plugin-for-gcp/pull/38) - [@stephanwehr](https://github.com/stephanwehr): [add support for incremental snapshots of Azure disks](https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/pull/52) # July 14, 2020 ### Status Updates - ashish - Community support last week - Fix for capturing version of CRD before version remap plugin: - PR 2683 (https://github.com/vmware-tanzu/velero/pull/2683) - Merged last week and is in this weeks [1.4.2 release](https://github.com/vmware-tanzu/velero/releases/tag/v1.4.2) - PR reviews: - VolumesnapshotProgress design proposal PR - merged last week. 🎉 - Ordered Resource backup design proposal - Couple of passes at the - Fixed [Issue 17](https://github.com/vmware-tanzu/velero-plugin-for-csi/issues/17) with [PR 60](https://github.com/vmware-tanzu/velero-plugin-for-csi/pull/60) in the [velero-plugin-for-csi repo](https://github.com/vmware-tanzu/velero-plugin-for-csi) - allows users of the CSI plugin to keep regular volumesnapshotclass separate from that used by Velero. Primarily to allow different `DeletionPolicy`. - [carlisia] - Community support - Blocker (BSL controller): https://github.com/vmware-tanzu/velero/pull/2674 - [nrb] - v1.4.2 is out! - Includes fixes from CRD - Going to try to help catch up on community support this week, particularly on GH issues/PRs. - Get master -> main transition done this week ### Discussion topics - Emmet - cacert integration w/ restic & rancher - https://github.com/vmware-tanzu/velero/issues/2675 ### Contributor Shoutouts - [@a-mccarthy](https://github.com/a-mccarthy): [add style guide file](https://github.com/vmware-tanzu/velero/pull/2619) - [@firethestars](https://github.com/firethestars): [Added Arm Multi-arch Make Logic](https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/36) - [@ehashman](https://github.com/ehashman): [Fix missing quotes that are breaking page render](https://github.com/vmware-tanzu/velero/pull/2698) # July 7, 2020 ### Status Updates - nrb - Community support last week, still playing catch up on GH issues though - Review blitz yesterday, but did not get them all - End in sight for our internal build process work, hope to give more time to this open source work! - Restore hooks meeting yesterday, thanks everyone for attending! - carlisia - BSL controller PR: https://github.com/vmware-tanzu/velero/pull/2674 - Worked on adding the BSL validation as webhooks; talked to @nrb and decided we won't do it - Did and doing PR reviews - Finished reviewing the volume backup/restore progress design doc - Next: converting ServerStatusRequest to controller-runtime - ashish - Community support this week: - PR to fix a CRD backup regression in `1.4`. [PR](https://github.com/vmware-tanzu/velero/pull/2683) [Issue](https://github.com/vmware-tanzu/velero/issues/2682) - Tested against `"v1.16.9" "v1.17.0" "v1.18.4"` - Will be included in the release for 1.4.1 - Working on internal build setup. Should mostly be done, need a few finishing touches - PR to fix the file permissions on the manifests in the backup tarball. - `0755` to `0644` - [PR 2685](https://github.com/vmware-tanzu/velero/pull/2685) ### Discussion topics - Phuong Demo of [Resource ordering](https://github.com/vmware-tanzu/velero/pull/2627) - nrb: `master` branch will be changing to `main`. We need to work on making sure docs redirects will work, but it looks like in-flight PRs will not be redirected unless GitHub introduces some sort of PR redirection feature. https://github.com/vmware-tanzu/velero/pull/2677 is tracking that work. - vitta: [PR 2543](https://github.com/vmware-tanzu/velero/pull/2543) ### Contributor Shoutouts - [@danielthrasher](https://github.com/danielthrasher): [updated acceptable values on cron schedule for day of the week from 0-7 to 0-6](https://github.com/vmware-tanzu/velero/pull/2676) - [@mauilion](https://github.com/mauilion) & team: [Builder Image handling](https://github.com/vmware-tanzu/velero/pull/2620) - [@tbatard](https://github.com/tbatard) & team: [Add linters](https://github.com/vmware-tanzu/velero/pull/2615) - Everyone who attended the restore hooks meeting yesterday! # June 30, 2020 ### Status Updates - carlisia - BSL controller to handle validation + update BSL status phase - PR ready for review (this changes the behavior of the Velero server): https://github.com/vmware-tanzu/velero/pull/2674/ - WIP: add webhook validation to BSL create/update - PR reviews owed - ashish - on community support last week - worked on some internal build setup and testing of the build artifacts - Opened this issue last week [Issue 2673](https://github.com/vmware-tanzu/velero/issues/2673) - Currently working on some load testing for restic - nrb - Was out for the end of last week for family health issues, didn't get everything I wanted done. - Reviews! - Updates on docs, getting Jason S up to speed for implementing new commands - On community support this week ### Discussion topics - [nrb] Moving code from backup & restore controllers out into the pkg/{backup/restore} directories Helps with kubebuilder/concurrency - [stephanie] Scheduling a backup and restore focus meeting Monday July 6th, more info to come here ### Contributor shoutouts Vishnu Itta for the volume snapshotter progress design doc!! Let's get that merged this week (for real!) Marc Campbell for the restore hooks design doc! Phuong for the work on the ordering design doc! # June 23, 2020 ### Status Updates - nrb - Lots of reviews, but sorry if I didn't get to yours - Lots of internal work too - carlisia - On community support this week - kubebuilder PR still getting reviews: https://github.com/vmware-tanzu/velero/pull/2561 - PR reviews - Adding tests to: Add BSL controller - https://github.com/vmware-tanzu/velero/pull/2617 - ashish is out - Worked on backing up volumes with restic by default ### Discussion Topics - nrb [Restore hooks](https://github.com/vmware-tanzu/velero/pull/2465) - let's have a seperate call for this to address some concerns about using init containers. - Main use case seems to be quiescing/unquiescing databases - Will need to use some sort of tool to send this to a backup PV, and a tool to take it from the backup PV to a runtime PV - nrb/carlisia - [Public/private APIs](https://github.com/vmware-tanzu/velero/issues/2654) - Let us know what you rely on! - This will be phased, not all at once - In 2.0, the current Go client for Velero will be replaced by a controller-runtime client. The informers/listers will be replaced by a client manager. - Vishnu (Enable restic by default - https://github.com/vmware-tanzu/velero/pull/2611) and (Progress on backup/restore by plugins - https://github.com/vmware-tanzu/velero/pull/2543) ### Contributor Shoutouts - [@adamrushuk](https://api.github.com/users/adamrushuk): [Update basic-install and release-instructions documentation](https://api.github.com/repos/vmware-tanzu/velero/pulls/2638) - [@JessicaGreben](https://api.github.com/users/JessicaGreben): [add a supported provider: Storj object storage](https://api.github.com/repos/vmware-tanzu/velero/pulls/2635) - [@adamrushuk](https://api.github.com/users/adamrushuk): [Update basic-install.md: Add windows cli installation option via chocolatey](https://api.github.com/repos/vmware-tanzu/velero/pulls/2629) - [@tbatard](https://api.github.com/users/tbatard): [Update Jekyll to 4.1.0](https://api.github.com/repos/vmware-tanzu/velero/pulls/2625) # June 16, 2020 ### Status Updates - steve: - community support & reviews - nrb: Mostly focused on internal build tooling last week Intending to catch up on community review this week - ashish: - Focussing on internal build setup - Finishing up on [PR 2611](https://github.com/vmware-tanzu/velero/pull/2611) - community support this week - carlisia - wip: add kubebuilder/controller-runtime - https://github.com/vmware-tanzu/velero/pull/2561 - wip: add BSL controller w/ controller-runtime - https://github.com/vmware-tanzu/velero/pull/2617 ### Discussion Topics - [nrb] [Deletion plugins](https://github.com/vmware-tanzu/velero/issues/2400)? - Would we want to take this opportunity to also do pre-backup plugins to? I don't have a use case right now, more asking because it's an opportunity to do so - Next steps: Design doc - Use labels, not configmap - [nrb] [Restore hooks design](https://github.com/vmware-tanzu/velero/pull/2465) should be close to merging. Maintainers, please take another pass! - [duffie] Linters for Velero, possibly also image handling - [Vishnu] Restic snapshot specific differential size and https://github.com/vmware-tanzu/velero/pull/2543 - Problem: How much object storage did a given backup consume? - Is there a way to know differential size from the previous snapshot? - This would help to prune object storage - Restic does deduplication; if I have 10 backups and I delete backup 6, how much do I get back? - Could we add a field to progress to tell report this? - Restic's format design documents: https://restic.readthedocs.io/en/latest/100_references.html#design - [tony] Update to Jekyll 4.1.0, https://github.com/vmware-tanzu/velero/pull/2625 ### Contributor Shoutouts

Example	Syntax
Header	# Header
Unordered List	- Unordered List
Ordered List	1. Ordered List
Todo List	- [ ] Todo List
Blockquote	> Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th	19^th^
H₂O	H~2~O
Inserted text	++Inserted text++
Marked text	==Marked text==
Link	[link text](https:// "title")
Image	![image alt](https:// "title")
`Code`	`Code`
`var i = 0;`	```javascript var i = 0; ```
	:smile:
Externals	{%youtube youtube_id %}
L^aT_eX	$L^aT_eX$
This is a alert area.	:::info This is a alert area. :::