--- tags: keda --- # KEDA Community Standups ## Feb 16th Meeting ### Proposed Agenda - Alibaba Reference Case (Tom) - Finalizing the blog post and checking with CNCF to publish it is up next - CNCF (Tom) - Started working on annual review - Incubation proposal is coming after that - Would it be useful if we would document all our integrations? (Tom) - Relates to https://github.com/kedacore/governance/issues/14 - Early exploration of KEDA for IoT ([thread](https://kubernetes.slack.com/archives/CKZJ36A5D/p1607004394421300)) ### Attendees (add yourself) - Jeff - Jonathan (golioth.io) - Shubham - Ritika - Sonia ## Feb 2nd Meeting ### Proposed Agenda - KEDA 2.1 is out! (Tom - https://github.com/kedacore/keda/releases/tag/v2.1.0) - Here are some highlights: - Introduction of ClusterTriggerAuthentication for cluster-wide trigger authentication - Introducing new InfluxDB, MongoDB & OpenStack Swift scaler - Improvements to AWS Cloudwatch, Azure Event Hub, Kafka & Redis scalers - Automatically determine the RabbitMQ protocol (when possible) - Support for Redis clusters - Performance improvements - Azure Functions users can use it soon (https://github.com/Azure/azure-functions-core-tools/pull/2406) - ARM support (Tom) - We can request an ARM machine on which we can run a self-hosted GH Actions runner - Request to be created - Tracking in https://github.com/kedacore/governance/issues/12 - Issue reports will be easier, we are in the GitHub Issue forms preview (Tom) - Example![](https://pbs.twimg.com/media/EtAg5h4XMAE9Nop?format=jpg&name=medium) - We'd 💘 some community help on our end-to-end tests ([open issues](https://github.com/kedacore/keda/issues?q=is%3Aissue+is%3Aopen+label%3Atesting)) ### Attendees (add yourself) - Tom Kerkhove (Codit) - Shubhuam Kuchhal (NEC) - Aaron Schlesinger (MSFT) - Jonathan Beri (Golioth) - Ahmed ElSayed (MSFT) - Jeff Hollan (MSFT) - Sonia Kulkarni (MSFT) ## Jan 19th Meeting ### Proposed Agenda - Support for ARM (https://github.com/kedacore/keda/issues/779) - we can produce images, but need an environment, that can be used to check the images (Z) - KEDA 2.1 (https://github.com/kedacore/keda/blob/main/CHANGELOG.md#unreleased) - we have a lot of improvements, fixes and new scalers, I think that we are good to go with a new release, once KEDA docs are up to date and the e2e tests are stabilized (Z) - KEDA 2.0 support in Azure Function CLI ([get started](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=windows%2Ccsharp%2Cbash)) - Available in [version >= 3.0.3216](https://github.com/Azure/azure-functions-core-tools/releases/tag/3.0.3216) - EKS, Calico, and CNI - and how we can support that - HTTP Add-on - CNCF Incubation ### Things to note - For ARM support we'd need an ARM machine to build, wondering if Google Cloud could provide with CNCF credits? Tom may check into this - Astronomer has a customer running in EKS, but they have replaced the EKS CNI and added Calico. Calico is handling all the traffic, but the control plane using CNI. - Pods that need to talk to control plane need to run on host network - Scaling events (calling Postgres for events), which needs access to the database via the network policies. - HTTP Add on https://github.com/kedacore/http-add-on/pull/2 - Hoping early / mid February for beta, Alpha sooner - Aaron open to writing a blog, ideally x-posting to https://keda.sh/blog/ ### Attendees - Jeff Hollan (MSFT) - Daniel Imberman (Astronomer.io) - Terry Humphries (GTP services) - Tom Kerkhove (Codit) - Aaron Schlesinger (MSFT) - Ritika (NEC) - Shubhuam Kuchhal (NEC) - Kris Dockery (Astronomer.io) - Jonathan Beri (Golioth) ### Notes - TODO ## Jan 5th Meeting ### Proposed Agenda - Support for cluster-scoped trigger authentication ([issue](https://github.com/kedacore/keda/issues/1469)) - E2E tests are unstable :/ - A lot of reported issues with EKS and Calico (networking related) ### Things to note - TODO ### Attendees - Jeff Hollan (MSFT) - Daniel Imberman (Astronomer) - Tom Kerkhove (Codit) - Tsuyoshi (MSFT) - Jonathan Beri (Golioth - Serverless IoT startup) - Zbynek Roubalik (Red Hat) ### Notes - TODO ## Dec 8th Meeting ### Proposed Agenda - HTTP-based autoscaling & SMI (Tom) - Experimental add-on is available in [our org](https://github.com/kedacore/http-add-on) - This is a starting point and we'll compare it with other tech such as KNative & OpenFaaS but you should not need to run Prometheus - Aaron created first PR, to be reviewed by Zbynek - Tom is working with [SMI](https://smi-spec.io/) people to have the metrics in other places such as Ingress, Service, etc. - [Ingress v2](https://kubernetes-sigs.github.io/service-apis/) is also very interested ### Things to note - This is our last standup of 2020 - Targetting January the soonest for KEDA 2.1 - [Governance](https://github.com/kedacore/governance) repo is now available ### Attendees - Jonathan Beri (Golioth - Serverless IoT startup) - Callum Pilton (IBM) - Jess McCreery (IBM) - Richard Coppen (IBM) - Ritika (NEC) - Shubham Kuchhal (NEC) - Tom Kerkhove (Codit) ### Notes - None ## Nov 24th Meeting ### Proposed Agenda - HTTP-based autoscaling (Tom) - Pause scaling option (Zbynek) - https://github.com/kedacore/keda/issues/944 ### Things to note - Merch is now available on https://merch.keda.sh ### Attendees - Zbynek - Tom - Shubham - Rory - Aron - Richard - Nisheeka ### Notes - HTTP based autoscaling - Should support multiple ingresses, Prometheus metrics can give us universality in some cases. - Design document is updated in the issue. - SMI: we could support if needed, it is been taken in the account in the proposal. So far nothing is tightly coupled with KEDA, scaler could be swapped by SMI, the same applies to Interceptor. - Tom will prepare a repo for HTTP add on - The scaler will be external one first - IBM e2e tests are in the progress, no issues - Pause scaling option - will use annotation/label to on ScaledObject/ScaledJob. KEDA will skip scaling for the target Deployment when the label is present (HPA will be probably removed), there will be option to specify the number of replicas on the target Deployment during this phase. ## Nov 10th Meeting ### Proposed Agenda - KEDA 2.0 - Roadmap Review ### Attendees - Ahmed - Jeff - Liam - Richard - Bimsara - Shubham - Zbynek ### Notes - Congrats on the KEDA 2.0 Release 🎉 - Reviewed roadmap, bubbled a few items up to the top of the list we'd consider for 2.1 - https://github.com/kedacore/keda/projects/3 ## Oct 29th Meeting ### Proposed Agenda - KEDA 2.0 Go live - Update on Helm issus - Standup Move (Tom) - Can we move it to 16:00 UTC? - Otherwise it will be hard for me to join in the future, which is ok (now is the time if you want to get rid of me 🙊) - HTTP and KEDA (Aaron MSFT) - Enable KEDA as a single source of truth on scaler metrics ([#1281](https://github.com/kedacore/keda/issues/1281) - Tom) - Mir and Shashik PR review ### Attendees - Jeff (MSFT) - Tom (Codit) - Aaron (MSFT) - Anirudh (MSFT) - Ikaika (Carvana) - James (Carvana) - Mir (Linqia) - Shashik (Linqia) - Ritika (NEC) - Shubham (NEC) - Zbynek (Red Hat) ### Notes - Making good progress on 2.0. Fixed memory leak in scaler. No blocking issues known. Could release on Nov 4 - https://github.com/kedacore/charts/issues/88 - https://docs.google.com/document/d/1vhT-wwEPPNImFmLZHGJxdKjUvKWk6ckeE7QCLSOJzLY/edit# for blog post that goes live on Nov 4 - @Jeff to work on Azure update to help amplify - @Tom to work on document and help to share plan with Slack / maintainers - Rename master to main, and merge v2 to main (and make a v1 branch). @Zbynek can help do this - Rename keda-docs to main - Moving standup on Nov 10 (tuesdays) at 16:00 UTC which is an hour earlier for folks who have daylight savings. - Aaron spent some time using KEDA, and wanting to enable HTTP scaling workloads - Discussed a design to set a minimum threshold, specifically for Kafka, but felt it was a useful pattern to build in a way other scalers could mirror this functionality - https://github.com/kedacore/keda/issues/692 ## Oct 15th Meeting ### Proposed Agenda - KEDA 2.0 Go live - Roadmap review - Daylight savings - IBM MQ - Merch (Tom) - What do you think of the current designs? They are pending approval. - Things I've asked: - Can we have a mix of 1 & 2, ie. have blue & gray in option 1 but mint in option 2? - Can we fill the lightningbolt with white to make it more clear - Azure Service Bus Sample Update (Tom) - Migrated to KEDA 2.0 - Pod identity tutorial on the way - Helm 2 support (Tom) - I still causes issues and there is a big push to migrate to 3.x (https://twitter.com/HelmPack/status/1314294233994493953) - I think we should remove it as part of KEDA 2.0 ### Attendees - Richard Coppen - Jess McCreery - Callum Pilton - Shubhuam Kuchhal - Jeff Hollan - Tsuyoshi - Anirudh Garg - Ahmed ElSayed - Zbynek Roubalik - Tom Kerkhove ### Notes - Have a few new PRs, also a report of ScaledJobs memory leak, looking to resolve and then do RC 2 - Tom created a roadmap https://github.com/kedacore/keda/projects/3 - Daylight savings, currently we meet at 1600 UTC, thinking is that starting Nov 1 we move standup to 1700 UTC - Also move to Tuesdays? - Start with the week of Nov 9 and meet on Nov 10 - IBM MQ scaler - https://developer.ibm.com/components/ibm-mq/series/badge-ibm-mq-developer-essentials - https://github.com/kedacore/keda/pull/1259 - Samples to v2 - Jeff to ping the samples to move to v2 - We are ok to remove helm 2 for the KEDA 2.0 charts given it's going EOL in November ### Merch Mockups **Option 1** ![](https://i.imgur.com/jnAP0Lr.jpg) **Option 2** ![](https://i.imgur.com/hfK1TKZ.jpg) ## Oct 1st Meeting ### Proposed Agenda - KEDA 2.0 Beta - Meeting notes (Tom) - Thoughts on this proposal? https://docs.google.com/document/d/1zdwD6j86GxcCe5S5ay9suCO77WPrEDnKSfuaI24EwM4/edit?usp=sharing - High availability (Jeff) - Roadmap (Tom) - Might be good if we define a longer-term roadmap for KEDA - It should be flexible so that we can easily pivot - Requested by community ### Attendees - Jeff Hollan (Microsoft) - Tsuyoshi (Microsoft) - Anirudh Garg (Microsoft) - Zbynek Roubalik (Red Hat) - Travis Bickford (Shutterstock) - Ritika Sharma (NEC) - Shubham Kuchhal (NEC) ### Notes - KEDA 2.0 beta going well, a few minor things to do before KEDA beta 2, Tsuyoshi working on some push stuff with scaled jobs (working to repro and fix) and some minor stuff in helm chart - Tom was able to do the Azure podcast about KEDA - Alibaba cloud was looking to do a blog post around how they are integrating with KEDA, and hopefully get logo on website - Travis has a PR for the helm chart that has the API CRD changes - though a bug in helm 3.3 with linting, and blocked on that for now. - Wanting to revert to older version of helm (< 3.3) for now so we can get it out there - Preference is whatever is easiest / most convenient - High availability - Current state is that we DO have readiness and liveness probes, so Kubernetes should detect that it is down if it locked up - The bottleneck currently is the metrics server. We can deploy multiple operators per namespace to partition in that regard, but are restricted to only have a single metrics server per cluster. - Relying on a library for K8s that doesn't allow us to deploy multiple metrics servers - If we did this it would require significant changes in the library - Other potential ineffeciency is that we open and close connection each polling interval. Could do some better pooling / connection re-use to reduce the amount of outbound connections we are making - https://github.com/kedacore/keda/issues/1121 - Best practice may be to do the multiple namespaces to partition it a bit - Should create roadmap - EVERYONE should take a bit of time to review some slack / issue stuff that's popped up around v2 - [ ] Jeff to add travis to have permission for GH actions on repos - [ ] Jeff and Tom to start building out a roadmap - [ ] Tom to help push out the KEDA 2.0 RC ## Sep 17th Meeting ### Proposed Agenda - KEDA 2.0 release - no major issues - Support for Helm 2.x - Travis has a solution for this, will follow up with him. - New Logo is in mockup stage (Tom) - Created an issue for CNCF incubation stage but a bit of work required for that (Tom) - Have to do some process level stuff - Pod Identity support in Azure Event Hubs - (Anirudh) to see if he can help get that implemented - Ask for Pod Identity support for Log Analytics - the person who implemented the Scaler for this will do it. - Since the v2 Beta is out - we need to decide how long should be wait before moving out of Beta - Tentatively **early October** if we dont hear any significant issues - Usage - Issues - If we dont hear anything significant her - Open Items - Question on how we construct the Scalers from the Golang code (Zbynek) - Better connection management for the event sources - (Ahmed) will have a look at it - https://github.com/kedacore/keda/issues/1121 - (Tom) is doing a podcast for KEDA next week - (Tom) We might be using some vulnerable packages in our testing - https://github.com/kedacore/keda/network/alerts - (Zbynek) can look at this ### Attendees - Zbynek - Ahmed - Tom - Anirudh ## Sept 3rd Meeting ### Proposed Agenda - Standup Notes (Tom) - Anyone having laggy experience lately as well? It's pretty unusable to me lately - Should we move to Google docs? Happy to setup if we need to. - Renaming metadata propertied to indicate source of the value (Tom - #1044) - Change looks good to me but I would not only rename it to awsAccessKeyIDFromEnv but also support: - Using literal values (awsAccessKeyID) - Using trigger authentication (awsAccessKeyID) - This gives people to use the right approach for their - Now is the time with v2 imo - Support for Helm 2.x (Tom) - Helm 2.0 is [officially end of support](https://helm.sh/blog/helm-v2-deprecation-timeline/) and [v2.16.10](https://github.com/helm/helm/releases/tag/v2.16.10) is the last release - I think we should stop supporting it so we can fix https://github.com/kedacore/charts/issues/18 - Docker Hub's new policy limiting image retention & Announcement for GitHub Container Registry (Tom - #995) - Docker Hub is applying throttling on pulls (user-level) and automatically removing image tags that are no longer used (6m) - GitHub Container Registry is announced which stores all images in our org - Do we move or not? What is the benefit? - I believe this will give us more metrics and centralizes everything in one place, but not a must - Personally, I would move the CIs to GH CR and explore what it looks like. After that, evaluate if we should move our official images with a migration period - Hacktoberfest is coming (Tom) - Any chores we can list for people to pick up? - KEDA artwork (missing lightning bolt) - https://github.com/kedacore/keda/issues/650 - Any update on the logo? - Standup Security - Do we need to any more action or did we set the password already @jeff? - Pod Identity support for Azure Event Hubs (Tom) - Any chance we can add this for 2.0? Ahmed? - https://github.com/kedacore/keda/issues/994 ### Attendees - Jeff - Zbynek - Travis - Tomasz - Anirudh ### Notes - Hackmd- Zbynek mentioned that sometimes it lags for him as well (it's doing it for me right now). We're cool with google docs - Just need to clean up formatting on the FromEnv and LGTM - I think the best course of action is take the Travis fix. Pending closure on when /if 2.x charts get published and how the repo is structured but that approach for Helm 2 support seems good. - On the docker registry one Jeff will email CNCF to see what other projects are doing. - Group action item to mark items with Hacktoberfest / Good first issu / help wanted. - Anirudh mentioned maybe we could put a banner on the keda.sh site that points to a simple docs page or even https://github.com/kedacore/keda/contribute - Provide some useful description in the issues with hacktoberfest so people know where to start. So we need to revie w all issues and add clear action items / what's expected. - Jeff will poke in jest the person who needed a few weeks before tweaking the SVG. In the meantime Tomasz mentioned he can give a shot - Jeff will ad a password this week - Event Hubs - v2, just need to merge PRs and the Helm chart and migration guide for scaled jobs. Also update the migration guide with the table of metadata name changes from Ahmed - Goal to release 2.x beta on TUESDAY ### Action Items - [ ] TODO ## Aug rd Meeting ### Proposed Agenda - 2.0 Release - Blog post for alpha is available for preview - https://github.com/kedacore/keda-docs/pull/226) - What are the dates for alpha/stable? - 2.0 Outstanding issues (Zbynek): - Rename ScaledObject.spec.scaleTargetRef.containerName - https://github.com/kedacore/keda/issues/1017 - Thoughts? - Remove confusing fields in Scalers metadata - https://github.com/kedacore/keda/issues/753#issuecomment-656224125 - Volunteer to implement this? - Script to convert existing ScaledObjects from v1 to v2 - https://github.com/kedacore/keda/issues/946 - In progress by Travis - IMHO doesn't have to be done for alpha - ScaledJob Migration guide - https://github.com/kedacore/keda-docs/issues/225 - Kafka Scaler fix - PR opened: https://github.com/kedacore/keda/pull/996 - Pod Identity support for Azure Event Hubs (Tom) - Any chance we can add this for 2.0? Ahmed? - https://github.com/kedacore/keda/issues/994 - KEDA Dashboard (Tom - Post 2.0) - Would love to bring [KEDA Dashboard](https://github.com/kedacore/dashboard) back to life and make it a scaling dashboard which could optionally even show cluster autoscaler and other non-KEDA things - A bit similar to https://github.com/dapr/dashboard - KEDA Extensibility (Tom - Post 2.0) - Would love to have this in 2.1: - [Emit Kuberentes Events on major KEDA events](https://github.com/kedacore/keda/issues/530) - [Provide CloudEvent to extend KEDA](https://github.com/kedacore/keda/issues/479) - KEDA artwork (missing lightning bolt) - https://github.com/kedacore/keda/issues/650 - issue opened for almost 6m - essential for merch/swag designs and production ### Attendees - TODO ### Notes - TODO ### Action Items - [ ] TODO ## Aug 20th Meeting ### Proposed Agenda - 2.0 Release - Blog post for alpha is available for preview - https://github.com/kedacore/keda-docs/pull/226) - What are the dates for alpha/stable? - 2.0 Outstanding issues (Zbynek): - Rename ScaledObject.spec.scaleTargetRef.containerName - https://github.com/kedacore/keda/issues/1017 - Thoughts? - Remove confusing fields in Scalers metadata - https://github.com/kedacore/keda/issues/753#issuecomment-656224125 - Volunteer to implement this? - Script to convert existing ScaledObjects from v1 to v2 - https://github.com/kedacore/keda/issues/946 - In progress by Travis - IMHO doesn't have to be done for alpha - ScaledJob Migration guide - https://github.com/kedacore/keda-docs/issues/225 - Kafka Scaler fix - PR opened: https://github.com/kedacore/keda/pull/996 - Pod Identity support for Azure Event Hubs (Tom) - Any chance we can add this for 2.0? Ahmed? - https://github.com/kedacore/keda/issues/994 - KEDA Dashboard (Tom - Post 2.0) - Would love to bring [KEDA Dashboard](https://github.com/kedacore/dashboard) back to life and make it a scaling dashboard which could optionally even show cluster autoscaler and other non-KEDA things - A bit similar to https://github.com/dapr/dashboard - KEDA Extensibility (Tom - Post 2.0) - Would love to have this in 2.1: - [Emit Kuberentes Events on major KEDA events](https://github.com/kedacore/keda/issues/530) - [Provide CloudEvent to extend KEDA](https://github.com/kedacore/keda/issues/479) - KEDA artwork (missing lightning bolt) - https://github.com/kedacore/keda/issues/650 - issue opened for almost 6m - essential for merch/swag designs and production ### Attendees - TODO ### Notes - TODO ### Action Items - [ ] TODO ## Aug 6th Meeting ### Proposed Agenda - 2.0 Release (T) - Let's not forget the config alignment - Alpha & full release planning? - Proposal - Alpha next week, full end of Aug if possible - Number of replicas remains the same after deleting ScaledObject (T - [#947](https://github.com/kedacore/keda/issues/947)) - Maintain last known state if prometheus is unavailable (Z - [#965](https://github.com/kedacore/keda/issues/965)) - Migrate to the new Kubebuilder based Operator SDK framework (Z - [#969](https://github.com/kedacore/keda/issues/969)) - Any volunteer to implement Script to do the migration from v1 to v2? (Z - [#946](https://github.com/kedacore/keda/issues/946)) - Standup authentication changes with Zoom (T - [#963](https://github.com/kedacore/keda/issues/963)) - Cluster Autoscale ### Attendees - Jeff Hollan - Zbynek Roubalik - Travis Bickford - Tsuyoshi - Patrick McQuighan - Tom Kerkhove (Codit) ### Notes - 2.0 release is almost done - Migration guide is on its way [link](https://keda.sh/docs/2.0/migration/) - Fix job related problems and ship alpha next week - Full release at end of the month, maybe sooner - Tom will start working on the blog post next week ([WIP](https://docs.google.com/document/d/1vhT-wwEPPNImFmLZHGJxdKjUvKWk6ckeE7QCLSOJzLY/edit?usp=sharing)) - We should try to get the samples up-to-date based on the alpha version and assign an owner or archive them - Number of replicas remains the same after deleting ScaledObject - People agree that it should scale back to the original number - However, Kubernetes & HPA without KEDA use the same approach - [ ] Tom will update the issue to keep current approach and give option to scale back to the original instance count - Maintain last known state if prometheus is unavailable - Prometheus metric adapter will maintain the same instance count, while KEDA will scale back to 0 if it doesnt' have the count - HPAs receiving errors from metric servers stop taking any action; this seems liek a safe aproach for us. Otherwise we can create autoscaling loops flooding whole cluster. - This is how it works today so we are good to go - We shouldn't do 1 -> 0 nor 0 -> 1 scaling if we cannot fetch metrics - Would be nice for 2.0, otherwise afterwards - Migrate to the new Kubebuilder based Operator SDK framework - We are using Operator SDK which is being merged with Kubebuilder, we should check if we can migrate - Has no functionality changes but just reshuffling of the code - Will be done for 2.0 stable release - Any volunteer to implement Script to do the migration from v1 to v2? - Travis will look at this - Aiming for full release - Standup authentication changes with Zoom - We need to migrate before Sept 21st - We'll use password for meetings - Cluster Autoscale - A few folks at Amazon are managing EKS and looking at KEDA for event-driven autoscaling - It won't be easy for cluster autoscaler as they don't provide APIs - Long-requested feature for Kubernetes upstream - Still in exploration phase - Related - https://github.com/kubernetes/autoscaler/issues/3182 - e2e tests - They seem flaky, what cluster are we running on? - Jeff will check with Ahmed - That's why we need cluster autoscaler integration ;) ## Archived Notes: - [May 9 2019 - Jul 23 2020](https://hackmd.io/@jeffhollan/SkqW9_FVP)