KSE v3.3.1 Changelog

# KSE v3.3.1 Changelog  - [KSE v3.3.1 Changelog](#kse-v331-changelog) - [Observability](#observability) - [Features](#features) - [Enhancements \& Updates](#enhancements--updates) - [Bug Fixes](#bug-fixes) - [DevOps](#devops) - [Features](#features-1) - [Enhancements \& Updates](#enhancements--updates-1) - [Bug Fixes](#bug-fixes-1) - [Multi-Tenancy \& Multi-Cluster](#multi-tenancy--multi-cluster) - [Features](#features-2) - [Bug Fixes](#bug-fixes-2) - [Microservice](#microservice) - [Features](#features-3) - [Storage](#storage) - [Features](#features-4) - [Bug Fixes](#bug-fixes-3) - [Authentication \& Authorization](#authentication--authorization) - [Features](#features-5) - [Enhancements \& Updates](#enhancements--updates-2) - [Bug Fixes](#bug-fixes-4) - [DMP](#dmp) - [Features](#features-6) - [App Store](#app-store) - [Features](#features-7) - [Bug Fixes](#bug-fixes-5) - [Network](#network) - [Enhancements \& Updates](#enhancements--updates-3) - [Bug Fixes](#bug-fixes-6) - [Service Mesh](#service-mesh) - [Bug Fixes](#bug-fixes-7) - [KubeEdge Integration](#kubeedge-integration) - [Features](#features-8) - [Enhancements \& Upgrades](#enhancements--upgrades) - [Bug Fixes](#bug-fixes-8) - [License](#license) - [Features](#features-9) - [Security](#security) - [Features](#features-10) - [Enhancements \& Updates](#enhancements--updates-4) - [Development \& Testing](#development--testing) - [Features](#features-11) - [User Experience](#user-experience) - [Features](#features-12) - [Enhancements \& Updates](#enhancements--updates-5) - [Bug Fixes](#bug-fixes-9) - [API Changes](#api-changes)  Starting from v3.3.1, the previous QKCP(QingCloud KubeSphere Container Management Platform) has been renamed to KSE (KubeSphere Container Management Platform Enterprise, or KubeSphere Enterprise for short) ## Observability ### Features - Add Whizard as the Prometheus Long-Term storage for multi-cluster monitoring & alerting ([Whizard](https://github.com/WhizardTelemetry/whizard) by [@junotx](https://github.com/junotx), [@wanjunlei](https://github.com/wanjunlei), and [@frezes](https://github.com/frezes)) - Add Observability Center ([kse-console#195](https://github.com/kubesphere/kse-console/pull/195), [@harrisonliu5](https://github.com/harrisonliu5)) - Support multi-cluster monitoring in the Observability Center ([kse#325](https://github.com/kubesphere/kse/pull/325), [@frezes](https://github.com/frezes)) - Support multi-cluster alerting in the Observability Center ([kse-console#267](https://github.com/kubesphere/kse-console/pull/267), [@weili520](https://github.com/weili520)) - Introduces 3 new CRDs GlobalRuleGroups, ClusterRuleGroups, and RuleGroups for multi-cluster, cluster, and project alerting respectively ([kse#297](https://github.com/kubesphere/kse/pull/297), [@junotx](https://github.com/junotx)) - Alert Rules are now defined in GlobalRuleGroups, ClusterRuleGroups or RuleGroups which will be converted to groups in PrometheusRules. - Add controller to sync PrometheusRules to host cluster ([kse#308](https://github.com/kubesphere/kse/pull/308), [@junotx](https://github.com/junotx)) - Add admission webhook and controllers for RuleGroups, ClusterRuleGroups, GlobalRuleGroups. ([kse#302](https://github.com/kubesphere/kse/pull/302), [@junotx](https://github.com/junotx)) - Built-in alert rules can be modified, enabled, disabled or reset now. - Support KubeSphere core components alerting ([kse-installer#65](https://github.com/kubesphere/kse-installer/pull/65), [@junotx](https://github.com/junotx)) - Support cleaning up a cluster's notification settings when removing this cluster ([kse-console#218](https://github.com/kubesphere/kse-console/pull/218), [@yazhouio](https://github.com/yazhouio)) - Add feishu notification channel ([kse-console#163](https://github.com/kubesphere/kse-console/pull/163), [@harrisonliu5](https://github.com/harrisonliu5)) - Support silencing notifications ([kse-console#205](https://github.com/kubesphere/kse-console/pull/205), [@yazhouio](https://github.com/yazhouio)) - Support switching notification language between English and Chinese ([kse-console#205](https://github.com/kubesphere/kse-console/pull/205), [@yazhouio](https://github.com/yazhouio)) - Support adding a user account's email address to this user's notification setting ([kse-console#165](https://github.com/kubesphere/kse-console/pull/165), [@harrisonliu5](https://github.com/harrisonliu5)) - Use OpenSearch as built-in log storage instead of Elasticsearch ([kse-installer#54](https://github.com/kubesphere/kse-installer/pull/54), [@wenchajun](https://github.com/wenchajun)) - Add OpenSearch curator ([kse-installer#54](https://github.com/kubesphere/kse-installer/pull/54), [@wenchajun](https://github.com/wenchajun)) - Add OpenSearch Dashboard ([kse-installer#79](https://github.com/kubesphere/kse-installer/pull/79), [@wenchajun](https://github.com/wenchajun)) - Support adding OpenSearch as log receiver ([kse-console#182](https://github.com/kubesphere/kse-console/pull/182), [@harrisonliu5](https://github.com/harrisonliu5)) - Support searching logs from OpenSearch in KubeSphere console ([kse#296](https://github.com/kubesphere/kse/pull/296) & [kse#400](https://github.com/kubesphere/kse/pull/400), [@wenchajun](https://github.com/wenchajun)) - Add process and thread monitoring metrics for containers. ([kubesphere#4711](https://github.com/kubesphere/kubesphere/pull/4711), [@junotx](https://github.com/junotx)) - Add disk monitoring metrics that provides disk usage data. ([kubesphere#4705](https://github.com/kubesphere/kubesphere/pull/4705), [@junotx](https://github.com/junotx)) - Support importing Grafana templates to create custom monitoring dashboards in a namespace. ([kubeshere#4446](https://github.com/kubesphere/kubesphere/pull/4446), [@zhu733756](https://github.com/zhu733756), [console#3202](https://github.com/kubesphere/console/pull/3202) [@weili520](https://github.com/weili520)) - Support defining data retention periods for container logs, K8s events log, and audit logs respectively. ([ks-installer#1915](https://github.com/kubesphere/ks-installer/pull/1915), [@wenchajun](https://github.com/wenchajun)) ### Enhancements & Updates - Cluster overview enhancement ([kse-console#177](https://github.com/kubesphere/kse-console/pull/177) by [@harrisonliu5](https://github.com/harrisonliu5), [kse#312](https://github.com/kubesphere/kse/pull/312) by [@frezes](https://github.com/frezes)) - Workspace & Project monitoring enhancement ([kse#337](https://github.com/kubesphere/kse/pull/337), [@frezes](https://github.com/frezes)) - `Alert Policies` has been renamed to `Rule Groups`, while `Alert Messages` has been renamed to `Alerts` - `Custom Alert Messages` has been renamed to `From Custom Rules`, while `Built-in Alert Messages` has been renamed to `From Built-in Rules` - Change metric query statements to match Prometheus component upgrades. ([kubesphere#4621](https://github.com/kubesphere/kubesphere/pull/4621), [@junotx](https://github.com/junotx)) - Upgrade promethues to v2.39.1, thanos to v0.29.5 (Customized), and kube-state-metrics to v2.6.0 ([kse-installer#86](https://github.com/kubesphere/kse-installer/pull/86), [@frezes](https://github.com/frezes)) - Upgrade fluentbit-operator to v0.14.0 ([kse-installer#81](https://github.com/kubesphere/kse-installer/pull/81), [@wenchajun](https://github.com/wenchajun)) - Upgrade Notification Manager to v2.1.0 - Upgrade Alertmanager from v0.21.0 to v0.23.0. - Upgrade Grafana from v7.4.3 to v8.3.3. - Upgrade node-exporter from v0.18.1 to v1.3.1. - Upgrade Prometheus Operator from v0.43.2 to v0.55.1. - Upgrade kube-rbac-proxy from v0.8.0 to v0.11.0. - Upgrade configmap-reload from v0.3.0 to v0.5.0. - Upgrade kube-events from v0.3.0 to v0.4.0. - Upgrade Fluent Bit from v1.8.3 to v1.8.11. ### Bug Fixes - Fix incorrect ingress P95 delay promql expression. ([kse#314](https://github.com/kubesphere/kse/pull/314), [@iawia002](https://github.com/iawia002)) - Fix the style of log detail in toolbox ([kse-console#184](https://github.com/kubesphere/kse-console/pull/184), [@harrisonliu5](https://github.com/harrisonliu5)) - Fix inaccurate time unit of the monitoring metrics.([console#3557](https://github.com/kubesphere/console/pull/3557), [@iawia002](https://github.com/iawia002)) - Fix the "No Cluster Available" issue during log search. ([console#3555](https://github.com/kubesphere/console/pull/3555), [@harrisonliu5](https://github.com/harrisonliu5)) ## DevOps ### Features - Add the Continuous Deployment feature, which supports GitOps and uses Argo CD as the backend. ([ks-devops#466](https://github.com/kubesphere/ks-devops/pull/466), [@LinuxSuRen](https://github.com/LinuxSuRen), [console#2990]( https://github.com/kubesphere/console/pull/2990),[@harrisonliu5](https://github.com/harrisonliu5)) - Add a webhook for processing pipeline events. ([ks-devops#442](https://github.com/kubesphere/ks-devops/pull/442), [@JohnNiang](https://github.com/JohnNiang)) - Add support for verifying SCM tokens directly instead of calling a Jenkins API. ([ks-devops#439](https://github.com/kubesphere/ks-devops/pull/439), [@LinuxSuRen](https://github.com/LinuxSuRen)) - Optimize pipeline templates loaded from APIs. ([ks-devops#453](https://github.com/kubesphere/ks-devops/pull/453), [@JohnNiang](https://github.com/JohnNiang), [console#2963](https://github.com/kubesphere/console/pull/2963), [@harrisonliu5](https://github.com/harrisonliu5)) - Add the allowlist feature on the **Basic Information** page of a DevOps project. ([console#3224](https://github.com/kubesphere/console/pull/3224), [@harrisonliu5](https://github.com/harrisonliu5)) - Add the **Events** tab on the pipeline run record page. ([console#3012](https://github.com/kubesphere/console/pull/3012), [@harrisonliu5](https://github.com/harrisonliu5)) - Prevent passing tokens directly to the DevOps SCM verification API to avoid security vulnerabilities. Instead, only secret names are passed to the API. ([console#2943](https://github.com/kubesphere/console/pull/2943), [@mangoGoForward](https://github.com/mangoGoForward)) - Add a message indicating that the S2I and B2I features currently do not support the containerd runtime. ([console#2734](https://github.com/kubesphere/console/pull/2734), [@weili520](https://github.com/weili520)) - Fix an issue in the **Import Code Repository** dialog box, where repositories and organizations are not completely displayed. ([console#3222](https://github.com/kubesphere/console/pull/3222), [@harrisonliu5](https://github.com/harrisonliu5)) ### Enhancements & Updates - Add support for editing the kubeconfig binding mode on the pipeline UI.([console#3864](https://github.com/kubesphere/console/pull/3864), [@harrisonliu5](https://github.com/harrisonliu5)) ### Bug Fixes - Fix an issue where Jenkins updates are not synchronized in real time. ([ks-devops#837](https://github.com/kubesphere/ks-devops/pull/837), [@chilianyi](https://github.com/chilianyi)) - Fix the cron expression check failure of Jenkins. ([ks-devops#784](https://github.com/kubesphere/ks-devops/pull/784), [@LinuxSuRen](https://github.com/LinuxSuRen)) - Fix an issue where users fail to check the CI/CD template.([ks-devops-helm-chart#80](https://github.com/kubesphere-sigs/ks-devops-helm-chart/pull/80), [@chilianyi](https://github.com/chilianyi)) - Remove the `Deprecated` tag from the CI/CD template and replace `kubernetesDeploy` with `kubeconfig binding` at the deployment phase.([ks-devops-helm-chart#81](https://github.com/kubesphere-sigs/ks-devops-helm-chart/pull/81), [@chilianyi](https://github.com/chilianyi)) - Fix an issue where pipeline parameters are not updated in time.([console#3864](https://github.com/kubesphere/console/pull/3864), [@harrisonliu5](https://github.com/harrisonliu5)) ## Multi-Tenancy & Multi-Cluster ### Features - Support grouping cluster by labels. ([kse-console#193](https://github.com/kubesphere/kse-console/pull/193) & [kse-console#200](https://github.com/kubesphere/kse-console/pull/200), [@yazhouio](https://github.com/yazhouio)) - Add cluster label controller and APIs. ([kse#313](https://github.com/kubesphere/kse/pull/313), [@iawia002](https://github.com/iawia002)) - Add a limit on the number of labels that can be added to a cluster ([kse-console#203](https://github.com/kubesphere/kse-console/pull/203), [@yazhouio](https://github.com/yazhouio)) - A cluster's kubeconfig can now be renewed automatically([kse#304](https://github.com/kubesphere/kse/pull/304), [@iawia002](https://github.com/iawia002)) - Add metrics for cluster certificate expiration, now notifications can be sent when the kubeconfig is about to expire in 7 days or 24 hours ([kse#328](https://github.com/kubesphere/kse/pull/328), [@iawia002](https://github.com/iawia002)) - Add remove cluster warning - Add a notification to remind users when the kubeconfig certificate of a cluster is about to expire. ([console#3038](https://github.com/kubesphere/console/pull/3038), [@harrisonliu5](https://github.com/harrisonliu5), [kubesphere#4584](https://github.com/kubesphere/kubesphere/pull/4584), [@iawia002](https://github.com/iawia002)) - Add the name of current cluster to the `kubesphere-config` configmap. ([kubesphere#4679](https://github.com/kubesphere/kubesphere/pull/4679), [@iawia002](https://github.com/iawia002)) - Add an API to update a cluster's kubeconfig. ([kubesphere#4562](https://github.com/kubesphere/kubesphere/pull/4562), [@iawia002](https://github.com/iawia002)) - Support cluster member management and cluster role management. ([console#3061](https://github.com/kubesphere/console/pull/3061), [@harrisonliu5](https://github.com/harrisonliu5)) ### Bug Fixes - Set the ready status to false when a cluster failed to join the federation control plane. ([kse#315](https://github.com/kubesphere/kse/pull/315), [@iawia002](https://github.com/iawia002)) ## Microservice ### Features - Support Spring Cloud ([kse-console#157](https://github.com/kubesphere/kse-console/pull/157) by [@xuliwenwenwen](https://github.com/xuliwenwenwen), [kse-console#159](https://github.com/kubesphere/kse-console/pull/159) by [@harrisonliu5](https://github.com/harrisonliu5), [kse-console#280](https://github.com/kubesphere/kse-console/pull/280) by [@weili520](https://github.com/weili520)) - Add Spring Cloud Operator ([spring-cloud-operator](https://github.com/kubesphere-sigs/spring-cloud-operator/), [@RolandMa1986](https://github.com/RolandMa1986) & [@hongzhouzi](https://github.com/hongzhouzi) & [@wansir](https://github.com/wansir)) - Support Spring Cloud configuration in Spring Cloud Operator ([spring-cloud-operator#3](https://github.com/kubesphere-sigs/spring-cloud-operator/pull/3), [@RolandMa1986](https://github.com/RolandMa1986)) - Support Spring Cloud Microservice in Spring Cloud Operator ([spring-cloud-operator#9](https://github.com/kubesphere-sigs/spring-cloud-operator/pull/9) & [spring-cloud-operator#35](https://github.com/kubesphere-sigs/spring-cloud-operator/pull/35), [@hongzhouzi](https://github.com/hongzhouzi)) - Support Spring Cloud Gateway in Spring Cloud Operator ([spring-cloud-operator#12](https://github.com/kubesphere-sigs/spring-cloud-operator/pull/12), [@wansir](https://github.com/wansir)) - Support Microservice Aggregated API in Spring Cloud Operator ([spring-cloud-operator#15](https://github.com/kubesphere-sigs/spring-cloud-operator/pull/15), [@hongzhouzi](https://github.com/hongzhouzi)) - Add GatewayRouteConfig in Spring Cloud Operator ([spring-cloud-operator#29](https://github.com/kubesphere-sigs/spring-cloud-operator/pull/29), [@wansir](https://github.com/wansir)) - Add Spring Cloud helm chart install ([kse-installer#58](https://github.com/kubesphere/kse-installer/pull/58) & [kse-installer#97](https://github.com/kubesphere/kse-installer/pull/97) & [kse-installer#73](https://github.com/kubesphere/kse-installer/pull/73), [@hongzhouzi](https://github.com/hongzhouzi)) - Add kubesphere-spring-cloud-starter ([spring-cloud-gateway#2](https://github.com/kubesphere-sigs/spring-cloud-gateway/pull/2), [@hongzhouzi](https://github.com/hongzhouzi)) ## Storage ### Features - Support PVC resizing that can automatically expand PVCs when the remaining capacity is insufficient. ([kubesphere#4660](https://github.com/kubesphere/kubesphere/pull/4660), [@f10atin9](https://github.com/f10atin9).[console#3056](https://github.com/kubesphere/console/pull/3056), [@weili520](https://github.com/weili520)) - Support volume snapshot content management and volume snapshot class management. ([kubesphere#4596](https://github.com/kubesphere/kubesphere/pull/4596), [@f10atin9](https://github.com/f10atin9), [console#2958](https://github.com/kubesphere/console/pull/2958), [@weili520](https://github.com/weili520), [console#3051](https://github.com/kubesphere/console/pull/3051), [@weili520](https://github.com/weili520)) - Allow users to set authorization rules for storage classes so that storage classes can be used only in specific projects. ([kubesphere#4770](https://github.com/kubesphere/kubesphere/pull/4770), [@f10atin9](https://github.com/f10atin9), [console#3069](https://github.com/kubesphere/console/pull/3069), [@weili520](https://github.com/weili520)) - Provide usage data of each disk. ([console#3063](https://github.com/kubesphere/console/pull/3063), [@harrisonliu5](https://github.com/harrisonliu5)) ### Bug Fixes - Set `hostpath` as a required option when users are mounting volumes. ([console#3478](https://github.com/kubesphere/console/pull/3478), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao)) - Update storageClass-accessor, so that deleting storage resources no longer requires accessor validation.([kubesphere#5271](https://github.com/kubesphere/kubesphere/pull/5271),[@f10atin9](https://github.com/f10atin9)) ## Authentication & Authorization ### Features - Support manually disabling and enabling users. ([kubesphere#4695](https://github.com/kubesphere/kubesphere/pull/4695), [@wansir](https://github.com/wansir), [console#3014](https://github.com/kubesphere/console/pull/3014), [@harrisonliu5](https://github.com/harrisonliu5)) ### Enhancements & Updates - Check user permissions to clusters when creating or updating workspace templates. ([kse#382](https://github.com/kubesphere/kse/pull/382), [@zhou1203](https://github.com/zhou1203)) - Add authorization control when patching the `workspacetemplates` ### Bug Fixes - Support filtering workspace roles using the LabelSelector. ([kubesphere#5162](https://github.com/kubesphere/kubesphere/pull/5162), [@zhou1203](https://github.com/zhou1203)) - Support customizing or randomly setting an initial amdin password.([ks-install#2067](https://github.com/kubesphere/ks-installer/pull/2067),[@pixiake](https://github.com/pixiake)) - Delete annotations in `role-template-manage-users`, `role-template-view-members` and `role-template-manage-roles`.([ks-install#2062](https://github.com/kubesphere/ks-installer/pull/2062),[@zhou1203](https://github.com/zhou1203)) - Fix an issue where `cluster-admin` cannot view and manage the configmap, secret, and service account.([ks-install#2082](https://github.com/kubesphere/ks-installer/pull/2082),[@zhou1203](https://github.com/zhou1203)) - Delete role `workspace-manager`.([ks-install#2094](https://github.com/kubesphere/ks-installer/pull/2094),[@zhou1203](https://github.com/zhou1203)) - Add role `platform-self-provisioner`. ([ks-install#2095](https://github.com/kubesphere/ks-installer/pull/2095),[@zhou1203](https://github.com/zhou1203)) - Delete role `users-manager`. ([ks-install#2105](https://github.com/kubesphere/ks-installer/pull/2105),[@zhou1203](https://github.com/zhou1203)) - Block `role-template-manage-groups`. ([ks-install#2122](https://github.com/kubesphere/ks-installer/pull/2122),[@zhou1203](https://github.com/zhou1203)) ## RadonDB DMP ### Features #### DMP Framework: - Add Redis Cluster, Redis Sentinal, MongoDB, ECK, Kafka, OpenSearch, and RadonDB Operator(s) ([kse-console#221](https://github.com/kubesphere/kse-console/pull/221),[@zhanglin-doudou](https://github.com/zhanglin-doudou)) - Update RadonDB Operator helm charts ([kse-installer#102](https://github.com/kubesphere/kse-installer/pull/102) & [kse-installer#112](https://github.com/kubesphere/kse-installer/pull/112), [@inksnw](https://github.com/inksnw)) - Add a new postgresql operator implementation instead of kse3.2.1 postgresql operator - Refactor monitoring, add application monitoring information #### ClickHouse: - Support ClickHouse kernel version 21.1.3.32 - Rich display of monitoring metrics - Support horizontal and vertical expansion - SVC-based cluster load balancing - High-availability based on multiple replicas - Fine-grained cluster state presentation - Disk expansion support qingcloud-csi - Compatible kubernetes api extension version v1 - Automatically create/delete/configure ZooKeeper cluster dependencies #### MySQL: - Support make S3 and NFS backup schedule together ([radondb-mysql-kubernetes#721](https://github.com/radondb/radondb-mysql-kubernetes/pull/721)) - Add webhook check for sidecar image ([radondb-mysql-kubernetes#731](https://github.com/radondb/radondb-mysql-kubernetes/pull/731)) - support the x.x.x.x:/dbbackup nfs address. ([radondb-mysql-kubernetes#717](https://github.com/radondb/radondb-mysql-kubernetes/pull/717)) - validate mysql image and mysqlversion on cluster create ([radondb-mysql-kubernetes#715](https://github.com/radondb/radondb-mysql-kubernetes/pull/715)) - Support running single node. ([radondb-mysql-kubernetes#584](https://github.com/radondb/radondb-mysql-kubernetes/pull/584)) - feat(e2e): add podchaos api and test cases ([radondb-mysql-kubernetes#697](https://github.com/radondb/radondb-mysql-kubernetes/pull/721)) - feat(backup)：Support fast backup ([radondb-mysql-kubernetes#693](https://github.com/radondb/radondb-mysql-kubernetes/pull/721)) - support MySQL configuration template ([radondb-mysql-kubernetes#656](https://github.com/radondb/radondb-mysql-kubernetes/pull/656)) - Support MySQL subversion ([radondb-mysql-kubernetes#647](https://github.com/radondb/radondb-mysql-kubernetes/pull/647)) - add disk usage check in nfs backup ([radondb-mysql-kubernetes#679](https://github.com/radondb/radondb-mysql-kubernetes/pull/679)) - api,backup: add printcolumn for Success ([radondb-mysql-kubernetes#670](https://github.com/radondb/radondb-mysql-kubernetes/pull/670)) - update go version to 1.17 ([radondb-mysql-kubernetes#677](https://github.com/radondb/radondb-mysql-kubernetes/pull/677)) - chart: Support the use of Helm to install MySQL cluster ([radondb-mysql-kubernetes#519](https://github.com/radondb/radondb-mysql-kubernetes/pull/519)) #### ECK: - Implement a Hot-Warm-Cold deployment architecture - Deploy Kibana to manage clusters in a visualized manner - Integrated Elasticsearch Exporter with Prometheus #### Kafka: - Support Kafka 3.1 and 3.2 - Install and manage Kafka and ZooKeeper nodes on the GUI - Support automatic cluster failover and recovery - Dynamically expand the cluster capacity and storage - Integrate with Prometheus to support the Exporter-based resource status monitoring feature - Support TLS to enhance the security of internal access - Deploy and manage users and topics and display the ACL on the GUI - Support cluster rebalancing on the GUI #### MongoDB: - Add the service monitoring feature of MongoDB clusters - Improve database account management - Support dynamic disk expansion - Optimize the elegant start and stop logic - Display more cluster states #### OpenSearch: - Support installation and management of OpenSearch, Dashboard, and Logstash - Support OpenSearch 2.3.0 - Dynamically increase the compute specifications and storage capacity of clusters - Dynamically adjust the number of cluster nodes - Perform a rolling update upon configurations change - Support the keystore feature - Integrate with Prometheus to support the Exporter-based resource status monitoring feature - Support TLS #### PostgreSQL: - Support PostgreSQL kernel versions 12.12, 13.8, 14.5, and 15.0 - Support PostGIS and Citus plugins - Support scale-in/scale-out and scale-up/scale-down - Support semi-sync streaming replication - Display extensive monitoring metrics #### Redis: - Build based on Redis 6.x and 7.x - Introduce Proxy for clusters - Support Redis Cluster failover and auto-recovery - Suport resource status monitoring based on the Prometheus Exporter - Support Kubernetes requests and limits - Support password and password-free settings for clusters - Support hot updates of Redis's management configuration parameters ## App Store ### Features - Support to manage helm releases created by the native helm command ([kse#317](https://github.com/kubesphere/kse/pull/317) by [@xyz-li](https://github.com/xyz-li), [kse-console#192](https://github.com/kubesphere/kse-console/pull/192) by [@harrisonliu5](https://github.com/harrisonliu5)) - Add new options in openpitrix. ([kse-installer#62](https://github.com/kubesphere/kse-installer/pull/62), [@xyz-li](https://github.com/xyz-li)) ### Bug Fixes - Fix a ks-controller-manager crash error caused by Helm controller NPE errors. ([kubesphere#4602](https://github.com/kubesphere/kubesphere/pull/4602), [@chaunceyjiang](https://github.com/chaunceyjiang)) - Fix an issue when updating traffic policy in traffic monitor page of an application. ([console#3836](https://github.com/kubesphere/console/pull/3836), [@mujinhuakai](https://github.com/mujinhuakai)) ## Network ### Enhancements & Updates - Integrate OpenELB with KubeSphere for exposing LoadBalancer services. ([console#2993](https://github.com/kubesphere/console/pull/2993), [@weili520](https://github.com/weili520)) ### Bug Fixes - Fix an issue where the gateway of a project is not deleted after the project is deleted. ([kubesphere#4626](https://github.com/kubesphere/kubesphere/pull/4626), [@RolandMa1986](https://github.com/RolandMa1986)) - Fix an issue during gateway log query, where the log query connection is not terminated after the query is complete. ([kubesphere#4927](https://github.com/kubesphere/kubesphere/pull/4927), [@qingwave](https://github.com/qingwave)) - Fix an issue where users fail to create routing rules in IPv6 and IPv4 dual-stack environments.([console#3604](https://github.com/kubesphere/console/pull/3604), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao)) - Update the OpenELB check URL. ([console#3550](https://github.com/kubesphere/console/pull/3550), [@harrisonliu5](https://github.com/harrisonliu5)) ## Service Mesh ### Bug Fixes - Resolve port conflicts of virtual services that use multiple protocols. ([kubesphere#4560](https://github.com/kubesphere/kubesphere/pull/4560), [@RolandMa1986](https://github.com/RolandMa1986)) ## KubeEdge Integration ### Features - Add support for logging in to common cluster nodes and edge nodes from the KubeSphere web console. ([kubesphere#4579](https://github.com/kubesphere/kubesphere/pull/4579), [@lynxcat](https://github.com/lynxcat), [console#2888](https://github.com/kubesphere/console/pull/2888), [@lynxcat](https://github.com/lynxcat)) ### Enhancements & Upgrades - Upgrade KubeEdge from v1.7.2 to v1.9.2. - Remove EdgeWatcher as KubeEdge v1.9.2 provides similar functions. ### Bug Fixes - Change the cluster module key from `kubeedge` to `edgeruntime`.([console#3548](https://github.com/kubesphere/console/pull/3548), [@harrisonliu5](https://github.com/harrisonliu5)) ## License ### Features - Add Cluster ID to the license control mechanism ([kse#291](https://github.com/kubesphere/kse/pull/291) by [@xyz-li](https://github.com/xyz-li), [kse-console#183](https://github.com/kubesphere/kse-console/pull/183) by [@harrisonliu5](https://github.com/harrisonliu5)) - Add license for Observability Center ([kse-console#258](https://github.com/kubesphere/kse-console/pull/258), [@harrisonliu5](https://github.com/harrisonliu5)) - Add license for Spring Cloud. - Add metrics and alert rules for the KubeSphere licence, notifications will be sent if the KubeSphere license is about to expire in less than 30 days ([kse#324](https://github.com/kubesphere/kse/pull/324) by [@xyz-li](https://github.com/xyz-li), [kse-installer#65](https://github.com/kubesphere/kse-installer/pull/65) by [@junotx](https://github.com/junotx)) - Support showing the License ID in the license page. ## Security ### Features - Add Ingress for ks-console to support HTTPS ([kse#327](https://github.com/kubesphere/kse/pull/327), [@iawia002](https://github.com/iawia002)) ### Enhancements & Updates - Enforce the password length to be at least 8 characters - Add the logic to check if the username and password are same - Disable password autofilling ([kse-console#190](https://github.com/kubesphere/kse-console/pull/190), [@harrisonliu5](https://github.com/harrisonliu5)) ## Development & Testing ### Features - Add the --controllers flag to ks-controller-manager, which allows users to select controllers to be enabled for resource usage reduction during debugging. ([kubesphere#4512](https://github.com/kubesphere/kubesphere/pull/4512), [@live77](https://github.com/live77)) - Prevent ks-apiserver and ks-controller-manager from restarting when the cluster configuration is changed. ([kubesphere#4659](https://github.com/kubesphere/kubesphere/pull/4659), [@x893675](https://github.com/x893675)) - Add an agent to report additional information about ks-apiserver and controller-manager in debugging mode. ([kubesphere#4928](https://github.com/kubesphere/kubesphere/pull/4928), [@xyz-li](https://github.com/xyz-li)) ## User Experience ### Features - Now events can be displayed for abnormal pods ([kse-console#196](https://github.com/kubesphere/kse-console/pull/196), [@yazhouio](https://github.com/yazhouio)) - Add in-memory cache in addition to the original Redis cache ([kse#359](https://github.com/kubesphere/kse/pull/359), [@zhou1203](https://github.com/zhou1203)) - Update product name and logo to KubeSphere Enterprise ([kse-console#208](https://github.com/kubesphere/kse-console/pull/208), [@harrisonliu5](https://github.com/harrisonliu5)) - Add the End User License Agreements. - Add EULA in password confirm ([kse-console#181](https://github.com/kubesphere/kse-console/pull/181), [@harrisonliu5](https://github.com/harrisonliu5)) - Add support for more languages on the KubeSphere web console. ([console#2782](https://github.com/kubesphere/console/pull/2782), [@xuliwenwenwen](https://github.com/xuliwenwenwen)) - Add the lifecycle management feature for containers. ([console#2940](https://github.com/kubesphere/console/pull/2940), [@harrisonliu5](https://github.com/harrisonliu5)) - Add support for creating container environment variables in batches from secrets and configmaps. ([console#3044](https://github.com/kubesphere/console/pull/3044), [@weili520](https://github.com/weili520)) - Add a message in the **Audit Log Search** dialog box, which prompts users to enable the audit logs feature. ([console#3062](https://github.com/kubesphere/console/pull/3062), [@harrisonliu5](https://github.com/harrisonliu5)) - Add data units in the **Create Storage Class** dialog box. ([console#3200](https://github.com/kubesphere/console/pull/3200), [@weili520](https://github.com/weili520)) - Add the cluster viewing permission to a user when the user adds a cluster. ([console#3296](https://github.com/kubesphere/console/pull/3296), [@harrisonliu5](https://github.com/harrisonliu5)) - Add a time range selector on the **Traffic Monitoring** tab page. ([console#2916]( https://github.com/kubesphere/console/pull/2916), [@weili520](https://github.com/weili520)) - Add a prompt to remind users to select a language or artifact type when users are building images. ([console#3534](https://github.com/kubesphere/console/pull/3534), [@harrisonliu5](https://github.com/harrisonliu5)) - Add support for changing the number of items displayed on each page of a table.([console#3486](https://github.com/kubesphere/console/pull/3486), [@weili520](https://github.com/weili520)) - Add support for batch stopping workloads. ([console#3497](https://github.com/kubesphere/console/pull/3497), [@harrisonliu5](https://github.com/harrisonliu5)) - Add the creator annotation to ensure information displayed on the pod detail page is consistent with other details pages. ([console#3820](https://github.com/kubesphere/console/pull/3820), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao)) - Add support for displaying the revision record when the workload type is `statefulsets` or `daemonsets`.([console#3819](https://github.com/kubesphere/console/pull/3819), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao)) - Add a support page. ([console#3832](https://github.com/kubesphere/console/pull/3832), [@yazhouio](https://github.com/yazhouio)) - Add support for duplicate name validation of containers. ([console#3559](https://github.com/kubesphere/console/pull/3559), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao)) - Add support for duplicate name validation of service names. ([console#3696](https://github.com/kubesphere/console/pull/3696), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao)) ### Enhancements & Updates - Support setting the default container registry ([kse-console#197](https://github.com/kubesphere/kse-console/pull/197), [@yazhouio](https://github.com/yazhouio)) - Support the shortcut dropdown of the relevent events of the pods. - Support the quick jumping to the destination page and setting the rows quantity showed of the page. - Support to hide external links in the console - Remove the limit to 100 of the cpu-percent of HPA. - Optimize the service details area on the **Service Topology** page. ([console#2945](https://github.com/kubesphere/console/pull/2945), [@tracer1023](https://github.com/tracer1023)) - Prevent passwords without uppercase letters set through the backend CLI. ([kubesphere#4481](https://github.com/kubesphere/kubesphere/pull/4481), [@live77](https://github.com/live77)) - Optimize the UI text of cluster, workspace, and project deletion. ([console#3004](https://github.com/kubesphere/console/pull/3004), [@weili520](https://github.com/weili520)) - Set the **Token** parameter on the webhook settings page as mandatory. ([console#2903](https://github.com/kubesphere/console/pull/2903), [@xuliwenwenwen](https://github.com/xuliwenwenwen)) - Improve user experience on the web console of member clusters. ([kubesphere#4721](https://github.com/kubesphere/kubesphere/pull/4721), [@iawia002](https://github.com/iawia002), [console#3031](https://github.com/kubesphere/console/pull/3031), [@weili520](https://github.com/weili520)) - Optimize the **Service Type** and **External Access** columns of the service list. ([console#3058](https://github.com/kubesphere/console/pull/3058),[@weili520](https://github.com/weili520)) - Change the abbreviation of "ReadOnlyMany" from "ROM" to "ROX". ([console#2751](https://github.com/kubesphere/console/pull/2751), [@123liubao](https://github.com/123liubao)) - Improve the registry verification API to resolve registry verification failures. ([kubesphere#4678](https://github.com/kubesphere/kubesphere/pull/4678), [@wansir](https://github.com/wansir)) - Optimize image building of KubeSphere and the console.([console#3610](https://github.com/kubesphere/console/pull/3610), [@zt1046656665](https://github.com/zt1046656665)) ### Bug Fixes - Fix an issue where statefulset creation fails when a volume is mounted to an init container. ([console#2730](https://github.com/kubesphere/console/pull/2730), [@weili520](https://github.com/weili520)) - Fix an app installation failure, which occurs when users click buttons too fast. ([console#2735](https://github.com/kubesphere/console/pull/2735), [@weili520](https://github.com/weili520)) - Fix an incorrect message displayed during workload creation, which indicates that resource requests have exceeded limits. ([console#2809](https://github.com/kubesphere/console/pull/2809) , [@weili520](https://github.com/weili520)) - Fix an issue where no data is displayed on the **Traffic Management** and **Tracing** tab pages in a multi-cluster project. ([console#3195](https://github.com/kubesphere/console/pull/3195), [@harrisonliu5](https://github.com/harrisonliu5)) - Fix an issue where data in the service details area on the **Service Topology** page is not updated automatically. ([console#3117](https://github.com/kubesphere/console/pull/3117),[@weili520](https://github.com/weili520)) - Fix an issue where existing settings are not displayed when a stateful service is edited. ([console#2845](https://github.com/kubesphere/console/pull/2845), [@weili520](https://github.com/weili520)) - Fix an issue where container probes are still displayed after they are deleted. ([console#3213](https://github.com/kubesphere/console/pull/3213), [@harrisonliu5](https://github.com/harrisonliu5)) - Fix incorrect container names in the service details area on the **Service Topology** page. ([console#3276](https://github.com/kubesphere/console/pull/3276), [@weili520](https://github.com/weili520)) - Fix the incorrect number of worker nodes on the **Cluster Nodes** page. ([console#3279](https://github.com/kubesphere/console/pull/3279), [@weili520](https://github.com/weili520)) - Fix a workspace API error caused by an incorrect cluster role. ([console#3297](https://github.com/kubesphere/console/pull/3297), [@harrisonliu5](https://github.com/harrisonliu5)) - Fix an incorrect error message displayed when kubeconfig is updated for a second time. ([console#3392](https://github.com/kubesphere/console/pull/3392), [@harrisonliu5](https://github.com/harrisonliu5)) - Fix goroutine leaks that occur when a container terminal is opened. ([kubesphere#4918](https://github.com/kubesphere/kubesphere/pull/4918), [@anhoder](https://github.com/anhoder)) - Fix unexpected ks-apiserver panic caused by an index out of range error of metrics API calling. ([kubesphere#4691](https://github.com/kubesphere/kubesphere/pull/4691), [@larryliuqing](https://github.com/larryliuqing)) - Fix an ks-apiserver crash caused by resource discovery failure. ([kubesphere#4835](https://github.com/kubesphere/kubesphere/pull/4835), [@wansir](https://github.com/wansir)) - Fix an incorrect private key type in kubeconfig information. ([kubesphere#4936](https://github.com/kubesphere/kubesphere/pull/4936), [@xyz-li](https://github.com/xyz-li)) - Fix an issue where the key is not display in LoadBalancer. ([console#3503](https://github.com/kubesphere/console/pull/3503), [@harrisonliu5](https://github.com/harrisonliu5)) - Fix an issue where no prompt is displayed when users do not fill in key and value in the LoadBalancer access mode.([console#3499](https://github.com/kubesphere/console/pull/3499), [@weili520](https://github.com/weili520)) - Fix an issue where the update time of a service is incorrect on the detail page.([console#3803](https://github.com/kubesphere/console/pull/3803), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao)) - Fix inaccurate prompt when users are adding an init container.([console#3561](https://github.com/kubesphere/console/pull/3561), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao)) - Fix an issue where an error occurs while users enter Chinese characters in secret. ([console#3774](https://github.com/kubesphere/console/pull/3774), [@moweiwei](https://github.com/moweiwei)) - Fix an issue where the total number of pages is incorrectly displayed.([kubesphere#5201](https://github.com/kubesphere/kubesphere/pull/5201), [@yongxingMa](https://github.com/yongxingMa)) - Fix an issue where the update time of an application is incorrect. ([console#3541](https://github.com/kubesphere/console/pull/3541), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao)) - Fix a 404 issue in Documentation. ([console#3484](https://github.com/kubesphere/console/pull/3433) [@PrajwalBorkar](https://github.com/PrajwalBorkar)) - Fix an issue where the status of a cluster remains true when the cluster fails to join Federation. ([kubesphere#5137](https://github.com/kubesphere/kubesphere/pull/5137), [@x893675](https://github.com/x893675)) - Fix an issue where traffic allocation fails in the canary release mode. ([console#3542](https://github.com/kubesphere/console/pull/3542), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao)) - Fix an issue where configurations do not take effect when users set the pod request to 0.([console#3827](https://github.com/kubesphere/console/pull/3827), [@harrisonliu5](https://github.com/harrisonliu5)) - Fix an issue where canary release goes wrong when multiple operating systems are selected.([console#3479](https://github.com/kubesphere/console/pull/3479), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao)) - Fix an issue where configmap configurations cannot be saved while users are creating a workload.([console#3416](https://github.com/kubesphere/console/pull/3416),[@weili520](https://github.com/weili520)) ## API Changes - Alerting API has been upgraded to v2beta1, and the v2alpha1 alerting API is deprecated ([kse#331](https://github.com/kubesphere/kse/pull/331), [@junotx](https://github.com/junotx)) - Expose kube-apiserver of the host cluster as a LoadBalancer service for member clusters to access. ([kubesphere#4528](https://github.com/kubesphere/kubesphere/pull/4528), [@lxm](https://github.com/lxm)) - Provide RESTful APIs for ClusterTemplate. ([ks-devops#468](https://github.com/kubesphere/ks-devops/pull/468), [@JohnNiang](https://github.com/JohnNiang)) - Provide template-related APIs. ([ks-devops#460](https://github.com/kubesphere/ks-devops/pull/460), [@JohnNiang](https://github.com/JohnNiang)) - Change the KubeEdge proxy service to `http://edgeservice.kubeedge.svc/api/`. ([kubesphere#4478](https://github.com/kubesphere/kubesphere/pull/4478), [@zhu733756](https://github.com/zhu733756)) - Change the patch type of `PatchWorkspaceTemplate` from `MergePatchType` to `JSONPatchType`.([kubesphere#5217](https://github.com/kubesphere/kubesphere/pull/5217), [@zhou1203](https://github.com/zhou1203))