---
# System prepended metadata

title: KSE v3.3.1 Changelog

---

# KSE v3.3.1 Changelog
<!-- TOC -->

- [KSE v3.3.1 Changelog](#kse-v331-changelog)
	- [Observability](#observability)
		- [Features](#features)
		- [Enhancements \& Updates](#enhancements--updates)
		- [Bug Fixes](#bug-fixes)
	- [DevOps](#devops)
		- [Features](#features-1)
		- [Enhancements \& Updates](#enhancements--updates-1)
		- [Bug Fixes](#bug-fixes-1)
	- [Multi-Tenancy \& Multi-Cluster](#multi-tenancy--multi-cluster)
		- [Features](#features-2)
		- [Bug Fixes](#bug-fixes-2)
	- [Microservice](#microservice)
		- [Features](#features-3)
	- [Storage](#storage)
		- [Features](#features-4)
		- [Bug Fixes](#bug-fixes-3)
	- [Authentication \& Authorization](#authentication--authorization)
		- [Features](#features-5)
		- [Enhancements \& Updates](#enhancements--updates-2)
		- [Bug Fixes](#bug-fixes-4)
	- [DMP](#dmp)
		- [Features](#features-6)
	- [App Store](#app-store)
		- [Features](#features-7)
		- [Bug Fixes](#bug-fixes-5)
	- [Network](#network)
		- [Enhancements \& Updates](#enhancements--updates-3)
		- [Bug Fixes](#bug-fixes-6)
	- [Service Mesh](#service-mesh)
		- [Bug Fixes](#bug-fixes-7)
	- [KubeEdge Integration](#kubeedge-integration)
		- [Features](#features-8)
		- [Enhancements \& Upgrades](#enhancements--upgrades)
		- [Bug Fixes](#bug-fixes-8)
	- [License](#license)
		- [Features](#features-9)
	- [Security](#security)
		- [Features](#features-10)
		- [Enhancements \& Updates](#enhancements--updates-4)
	- [Development \& Testing](#development--testing)
		- [Features](#features-11)
	- [User Experience](#user-experience)
		- [Features](#features-12)
		- [Enhancements \& Updates](#enhancements--updates-5)
		- [Bug Fixes](#bug-fixes-9)
	- [API Changes](#api-changes)

<!-- /TOC -->

Starting from v3.3.1, the previous QKCP(QingCloud KubeSphere Container Management Platform) has been renamed to KSE (KubeSphere Container Management Platform Enterprise, or KubeSphere Enterprise for short)

## Observability

### Features

- Add Whizard as the Prometheus Long-Term storage for multi-cluster monitoring & alerting ([Whizard](https://github.com/WhizardTelemetry/whizard) by [@junotx](https://github.com/junotx), [@wanjunlei](https://github.com/wanjunlei), and [@frezes](https://github.com/frezes))
- Add Observability Center ([kse-console#195](https://github.com/kubesphere/kse-console/pull/195), [@harrisonliu5](https://github.com/harrisonliu5))
- Support multi-cluster monitoring in the Observability Center ([kse#325](https://github.com/kubesphere/kse/pull/325), [@frezes](https://github.com/frezes))
- Support multi-cluster alerting in the Observability Center ([kse-console#267](https://github.com/kubesphere/kse-console/pull/267), [@weili520](https://github.com/weili520))
- Introduces 3 new CRDs GlobalRuleGroups, ClusterRuleGroups, and RuleGroups for multi-cluster, cluster, and project alerting respectively ([kse#297](https://github.com/kubesphere/kse/pull/297), [@junotx](https://github.com/junotx))
- Alert Rules are now defined in GlobalRuleGroups, ClusterRuleGroups or RuleGroups which will be converted to groups in PrometheusRules.
- Add controller to sync PrometheusRules to host cluster ([kse#308](https://github.com/kubesphere/kse/pull/308), [@junotx](https://github.com/junotx))
- Add admission webhook and controllers for RuleGroups, ClusterRuleGroups, GlobalRuleGroups. ([kse#302](https://github.com/kubesphere/kse/pull/302), [@junotx](https://github.com/junotx))
- Built-in alert rules can be modified, enabled, disabled or reset now.
- Support KubeSphere core components alerting ([kse-installer#65](https://github.com/kubesphere/kse-installer/pull/65), [@junotx](https://github.com/junotx))
- Support cleaning up a cluster's notification settings when removing this cluster ([kse-console#218](https://github.com/kubesphere/kse-console/pull/218), [@yazhouio](https://github.com/yazhouio))
- Add feishu notification channel ([kse-console#163](https://github.com/kubesphere/kse-console/pull/163), [@harrisonliu5](https://github.com/harrisonliu5))
- Support silencing notifications ([kse-console#205](https://github.com/kubesphere/kse-console/pull/205), [@yazhouio](https://github.com/yazhouio))
- Support switching notification language between English and Chinese ([kse-console#205](https://github.com/kubesphere/kse-console/pull/205), [@yazhouio](https://github.com/yazhouio))
- Support adding a user account's email address to this user's notification setting ([kse-console#165](https://github.com/kubesphere/kse-console/pull/165), [@harrisonliu5](https://github.com/harrisonliu5))
- Use OpenSearch as built-in log storage instead of Elasticsearch ([kse-installer#54](https://github.com/kubesphere/kse-installer/pull/54), [@wenchajun](https://github.com/wenchajun))
- Add OpenSearch curator ([kse-installer#54](https://github.com/kubesphere/kse-installer/pull/54), [@wenchajun](https://github.com/wenchajun))
- Add OpenSearch Dashboard ([kse-installer#79](https://github.com/kubesphere/kse-installer/pull/79), [@wenchajun](https://github.com/wenchajun))
- Support adding OpenSearch as log receiver ([kse-console#182](https://github.com/kubesphere/kse-console/pull/182), [@harrisonliu5](https://github.com/harrisonliu5))
- Support searching logs from OpenSearch in KubeSphere console ([kse#296](https://github.com/kubesphere/kse/pull/296) & [kse#400](https://github.com/kubesphere/kse/pull/400), [@wenchajun](https://github.com/wenchajun))
- Add process and thread monitoring metrics for containers. ([kubesphere#4711](https://github.com/kubesphere/kubesphere/pull/4711), [@junotx](https://github.com/junotx))
- Add disk monitoring metrics that provides disk usage data. ([kubesphere#4705](https://github.com/kubesphere/kubesphere/pull/4705), [@junotx](https://github.com/junotx))
- Support importing Grafana templates to create custom monitoring dashboards in a namespace. ([kubeshere#4446](https://github.com/kubesphere/kubesphere/pull/4446), [@zhu733756](https://github.com/zhu733756), [console#3202](https://github.com/kubesphere/console/pull/3202) [@weili520](https://github.com/weili520))
- Support defining data retention periods for container logs, K8s events log, and audit logs respectively. ([ks-installer#1915](https://github.com/kubesphere/ks-installer/pull/1915), [@wenchajun](https://github.com/wenchajun))

### Enhancements & Updates

- Cluster overview enhancement ([kse-console#177](https://github.com/kubesphere/kse-console/pull/177) by [@harrisonliu5](https://github.com/harrisonliu5), [kse#312](https://github.com/kubesphere/kse/pull/312) by [@frezes](https://github.com/frezes))
- Workspace & Project monitoring enhancement ([kse#337](https://github.com/kubesphere/kse/pull/337), [@frezes](https://github.com/frezes))
- `Alert Policies` has been renamed to `Rule Groups`, while `Alert Messages` has been renamed to `Alerts`
- `Custom Alert Messages` has been renamed to `From Custom Rules`, while `Built-in Alert Messages` has been renamed to `From Built-in Rules`
- Change metric query statements to match Prometheus component upgrades. ([kubesphere#4621](https://github.com/kubesphere/kubesphere/pull/4621), [@junotx](https://github.com/junotx))
- Upgrade promethues to v2.39.1, thanos to v0.29.5 (Customized), and kube-state-metrics to v2.6.0 ([kse-installer#86](https://github.com/kubesphere/kse-installer/pull/86), [@frezes](https://github.com/frezes))
- Upgrade fluentbit-operator to  v0.14.0 ([kse-installer#81](https://github.com/kubesphere/kse-installer/pull/81), [@wenchajun](https://github.com/wenchajun))
- Upgrade Notification Manager to v2.1.0
- Upgrade Alertmanager from v0.21.0 to v0.23.0.
- Upgrade Grafana from v7.4.3 to v8.3.3.
- Upgrade node-exporter from v0.18.1 to v1.3.1.
- Upgrade Prometheus Operator from v0.43.2 to v0.55.1.
- Upgrade kube-rbac-proxy from v0.8.0 to v0.11.0.
- Upgrade configmap-reload from v0.3.0 to v0.5.0.
- Upgrade kube-events from v0.3.0 to v0.4.0.
- Upgrade Fluent Bit from v1.8.3 to v1.8.11.

### Bug Fixes

- Fix incorrect ingress P95 delay promql expression. ([kse#314](https://github.com/kubesphere/kse/pull/314), [@iawia002](https://github.com/iawia002))
- Fix the style of log detail in toolbox ([kse-console#184](https://github.com/kubesphere/kse-console/pull/184), [@harrisonliu5](https://github.com/harrisonliu5))
- Fix inaccurate time unit of the monitoring metrics.([console#3557](https://github.com/kubesphere/console/pull/3557), [@iawia002](https://github.com/iawia002))
- Fix the "No Cluster Available" issue during log search. ([console#3555](https://github.com/kubesphere/console/pull/3555), [@harrisonliu5](https://github.com/harrisonliu5))

## DevOps

### Features

- Add the Continuous Deployment feature, which supports GitOps and uses Argo CD as the backend. ([ks-devops#466](https://github.com/kubesphere/ks-devops/pull/466), [@LinuxSuRen](https://github.com/LinuxSuRen), [console#2990]( https://github.com/kubesphere/console/pull/2990),[@harrisonliu5](https://github.com/harrisonliu5))
- Add a webhook for processing pipeline events. ([ks-devops#442](https://github.com/kubesphere/ks-devops/pull/442), [@JohnNiang](https://github.com/JohnNiang))
- Add support for verifying SCM tokens directly instead of calling a Jenkins API. ([ks-devops#439](https://github.com/kubesphere/ks-devops/pull/439), [@LinuxSuRen](https://github.com/LinuxSuRen))
- Optimize pipeline templates loaded from APIs. ([ks-devops#453](https://github.com/kubesphere/ks-devops/pull/453), [@JohnNiang](https://github.com/JohnNiang), [console#2963](https://github.com/kubesphere/console/pull/2963), [@harrisonliu5](https://github.com/harrisonliu5))
- Add the allowlist feature on the **Basic Information** page of a DevOps project. ([console#3224](https://github.com/kubesphere/console/pull/3224), [@harrisonliu5](https://github.com/harrisonliu5))
- Add the **Events** tab on the pipeline run record page. ([console#3012](https://github.com/kubesphere/console/pull/3012), [@harrisonliu5](https://github.com/harrisonliu5))
- Prevent passing tokens directly to the DevOps SCM verification API to avoid security vulnerabilities. Instead, only secret names are passed to the API. ([console#2943](https://github.com/kubesphere/console/pull/2943), [@mangoGoForward](https://github.com/mangoGoForward))
- Add a message indicating that the S2I and B2I features currently do not support the containerd runtime. ([console#2734](https://github.com/kubesphere/console/pull/2734), [@weili520](https://github.com/weili520))
- Fix an issue in the **Import Code Repository** dialog box, where repositories and organizations are not completely displayed. ([console#3222](https://github.com/kubesphere/console/pull/3222), [@harrisonliu5](https://github.com/harrisonliu5))

### Enhancements & Updates
- Add support for editing the kubeconfig binding mode on the pipeline UI.([console#3864](https://github.com/kubesphere/console/pull/3864), [@harrisonliu5](https://github.com/harrisonliu5))

### Bug Fixes
- Fix an issue where Jenkins updates are not synchronized in real time. ([ks-devops#837](https://github.com/kubesphere/ks-devops/pull/837), [@chilianyi](https://github.com/chilianyi))
- Fix the cron expression check failure of Jenkins. ([ks-devops#784](https://github.com/kubesphere/ks-devops/pull/784), [@LinuxSuRen](https://github.com/LinuxSuRen))
- Fix an issue where users fail to check the CI/CD template.([ks-devops-helm-chart#80](https://github.com/kubesphere-sigs/ks-devops-helm-chart/pull/80), [@chilianyi](https://github.com/chilianyi))
- Remove the `Deprecated` tag from the CI/CD template and replace `kubernetesDeploy` with `kubeconfig binding` at the deployment phase.([ks-devops-helm-chart#81](https://github.com/kubesphere-sigs/ks-devops-helm-chart/pull/81), [@chilianyi](https://github.com/chilianyi))
- Fix an issue where pipeline parameters are not updated in time.([console#3864](https://github.com/kubesphere/console/pull/3864), [@harrisonliu5](https://github.com/harrisonliu5))

## Multi-Tenancy & Multi-Cluster

### Features

- Support grouping cluster by labels. ([kse-console#193](https://github.com/kubesphere/kse-console/pull/193) & [kse-console#200](https://github.com/kubesphere/kse-console/pull/200), [@yazhouio](https://github.com/yazhouio))
- Add cluster label controller and APIs. ([kse#313](https://github.com/kubesphere/kse/pull/313), [@iawia002](https://github.com/iawia002))
- Add a limit on the number of labels that can be added to a cluster ([kse-console#203](https://github.com/kubesphere/kse-console/pull/203), [@yazhouio](https://github.com/yazhouio))
- A cluster's kubeconfig can now be renewed automatically([kse#304](https://github.com/kubesphere/kse/pull/304), [@iawia002](https://github.com/iawia002))
- Add metrics for cluster certificate expiration, now notifications can be sent when the kubeconfig is about to expire in 7 days or 24 hours ([kse#328](https://github.com/kubesphere/kse/pull/328), [@iawia002](https://github.com/iawia002))
- Add remove cluster warning
- Add a notification to remind users when the kubeconfig certificate of a cluster is about to expire. ([console#3038](https://github.com/kubesphere/console/pull/3038), [@harrisonliu5](https://github.com/harrisonliu5), [kubesphere#4584](https://github.com/kubesphere/kubesphere/pull/4584), [@iawia002](https://github.com/iawia002))
- Add the name of current cluster to the `kubesphere-config` configmap. ([kubesphere#4679](https://github.com/kubesphere/kubesphere/pull/4679), [@iawia002](https://github.com/iawia002))
- Add an API to update a cluster's kubeconfig. ([kubesphere#4562](https://github.com/kubesphere/kubesphere/pull/4562), [@iawia002](https://github.com/iawia002))
- Support cluster member management and cluster role management. ([console#3061](https://github.com/kubesphere/console/pull/3061), [@harrisonliu5](https://github.com/harrisonliu5))

### Bug Fixes

- Set the ready status to false when a cluster failed to join the federation control plane. ([kse#315](https://github.com/kubesphere/kse/pull/315), [@iawia002](https://github.com/iawia002))

## Microservice

### Features

- Support Spring Cloud ([kse-console#157](https://github.com/kubesphere/kse-console/pull/157) by [@xuliwenwenwen](https://github.com/xuliwenwenwen), [kse-console#159](https://github.com/kubesphere/kse-console/pull/159) by [@harrisonliu5](https://github.com/harrisonliu5), [kse-console#280](https://github.com/kubesphere/kse-console/pull/280) by [@weili520](https://github.com/weili520))
- Add Spring Cloud Operator ([spring-cloud-operator](https://github.com/kubesphere-sigs/spring-cloud-operator/),  [@RolandMa1986](https://github.com/RolandMa1986) & [@hongzhouzi](https://github.com/hongzhouzi) & [@wansir](https://github.com/wansir))
- Support Spring Cloud configuration in Spring Cloud Operator ([spring-cloud-operator#3](https://github.com/kubesphere-sigs/spring-cloud-operator/pull/3),  [@RolandMa1986](https://github.com/RolandMa1986))
- Support Spring Cloud Microservice in Spring Cloud Operator ([spring-cloud-operator#9](https://github.com/kubesphere-sigs/spring-cloud-operator/pull/9) & [spring-cloud-operator#35](https://github.com/kubesphere-sigs/spring-cloud-operator/pull/35),  [@hongzhouzi](https://github.com/hongzhouzi))
- Support Spring Cloud Gateway in Spring Cloud Operator ([spring-cloud-operator#12](https://github.com/kubesphere-sigs/spring-cloud-operator/pull/12),  [@wansir](https://github.com/wansir))
- Support Microservice Aggregated API in Spring Cloud Operator ([spring-cloud-operator#15](https://github.com/kubesphere-sigs/spring-cloud-operator/pull/15),   [@hongzhouzi](https://github.com/hongzhouzi))
- Add GatewayRouteConfig in Spring Cloud Operator ([spring-cloud-operator#29](https://github.com/kubesphere-sigs/spring-cloud-operator/pull/29),  [@wansir](https://github.com/wansir))
- Add Spring Cloud helm chart install ([kse-installer#58](https://github.com/kubesphere/kse-installer/pull/58) & [kse-installer#97](https://github.com/kubesphere/kse-installer/pull/97) & [kse-installer#73](https://github.com/kubesphere/kse-installer/pull/73),  [@hongzhouzi](https://github.com/hongzhouzi))
- Add kubesphere-spring-cloud-starter ([spring-cloud-gateway#2](https://github.com/kubesphere-sigs/spring-cloud-gateway/pull/2),  [@hongzhouzi](https://github.com/hongzhouzi))

## Storage

### Features

- Support PVC resizing that can automatically expand PVCs when the remaining capacity is insufficient. ([kubesphere#4660](https://github.com/kubesphere/kubesphere/pull/4660), [@f10atin9](https://github.com/f10atin9).[console#3056](https://github.com/kubesphere/console/pull/3056), [@weili520](https://github.com/weili520))
- Support volume snapshot content management and volume snapshot class management. ([kubesphere#4596](https://github.com/kubesphere/kubesphere/pull/4596), [@f10atin9](https://github.com/f10atin9), [console#2958](https://github.com/kubesphere/console/pull/2958), [@weili520](https://github.com/weili520), [console#3051](https://github.com/kubesphere/console/pull/3051), [@weili520](https://github.com/weili520))
- Allow users to set authorization rules for storage classes so that storage classes can be used only in specific projects. ([kubesphere#4770](https://github.com/kubesphere/kubesphere/pull/4770), [@f10atin9](https://github.com/f10atin9), [console#3069](https://github.com/kubesphere/console/pull/3069), [@weili520](https://github.com/weili520))
- Provide usage data of each disk. ([console#3063](https://github.com/kubesphere/console/pull/3063), [@harrisonliu5](https://github.com/harrisonliu5))

### Bug Fixes
- Set `hostpath` as a required option when users are mounting volumes. ([console#3478](https://github.com/kubesphere/console/pull/3478), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao))
- Update storageClass-accessor, so that deleting storage resources no longer requires accessor validation.([kubesphere#5271](https://github.com/kubesphere/kubesphere/pull/5271),[@f10atin9](https://github.com/f10atin9))

## Authentication & Authorization

### Features
- Support manually disabling and enabling users. ([kubesphere#4695](https://github.com/kubesphere/kubesphere/pull/4695), [@wansir](https://github.com/wansir), [console#3014](https://github.com/kubesphere/console/pull/3014), [@harrisonliu5](https://github.com/harrisonliu5))

### Enhancements & Updates
- Check user permissions to clusters when creating or updating workspace templates. ([kse#382](https://github.com/kubesphere/kse/pull/382), [@zhou1203](https://github.com/zhou1203))
- Add authorization control when patching the `workspacetemplates`

### Bug Fixes
- Support filtering workspace roles using the LabelSelector. ([kubesphere#5162](https://github.com/kubesphere/kubesphere/pull/5162), [@zhou1203](https://github.com/zhou1203))
- Support customizing or randomly setting an initial amdin password.([ks-install#2067](https://github.com/kubesphere/ks-installer/pull/2067),[@pixiake](https://github.com/pixiake))
- Delete annotations in `role-template-manage-users`, `role-template-view-members` and `role-template-manage-roles`.([ks-install#2062](https://github.com/kubesphere/ks-installer/pull/2062),[@zhou1203](https://github.com/zhou1203))
- Fix an issue where `cluster-admin` cannot view and manage the configmap, secret, and service account.([ks-install#2082](https://github.com/kubesphere/ks-installer/pull/2082),[@zhou1203](https://github.com/zhou1203))
- Delete role `workspace-manager`.([ks-install#2094](https://github.com/kubesphere/ks-installer/pull/2094),[@zhou1203](https://github.com/zhou1203))
- Add role `platform-self-provisioner`. ([ks-install#2095](https://github.com/kubesphere/ks-installer/pull/2095),[@zhou1203](https://github.com/zhou1203))
- Delete role `users-manager`. ([ks-install#2105](https://github.com/kubesphere/ks-installer/pull/2105),[@zhou1203](https://github.com/zhou1203))
- Block `role-template-manage-groups`. ([ks-install#2122](https://github.com/kubesphere/ks-installer/pull/2122),[@zhou1203](https://github.com/zhou1203))

## RadonDB DMP
### Features

#### DMP Framework:

- Add Redis Cluster, Redis Sentinal, MongoDB, ECK, Kafka, OpenSearch, and RadonDB Operator(s) ([kse-console#221](https://github.com/kubesphere/kse-console/pull/221),[@zhanglin-doudou](https://github.com/zhanglin-doudou)) 
- Update RadonDB Operator helm charts ([kse-installer#102](https://github.com/kubesphere/kse-installer/pull/102) & [kse-installer#112](https://github.com/kubesphere/kse-installer/pull/112), [@inksnw](https://github.com/inksnw))
- Add a new postgresql operator implementation instead of kse3.2.1 postgresql operator
- Refactor monitoring, add application monitoring information

#### ClickHouse:

- Support ClickHouse kernel version 21.1.3.32
- Rich display of monitoring metrics
- Support horizontal and vertical expansion
- SVC-based cluster load balancing
- High-availability based on multiple replicas
- Fine-grained cluster state presentation
- Disk expansion support qingcloud-csi
- Compatible kubernetes api extension version v1
- Automatically create/delete/configure ZooKeeper cluster dependencies

#### MySQL:

- Support make S3 and NFS backup schedule together  ([radondb-mysql-kubernetes#721](https://github.com/radondb/radondb-mysql-kubernetes/pull/721))
- Add webhook check for sidecar image ([radondb-mysql-kubernetes#731](https://github.com/radondb/radondb-mysql-kubernetes/pull/731))
- support the x.x.x.x:/dbbackup nfs address.  ([radondb-mysql-kubernetes#717](https://github.com/radondb/radondb-mysql-kubernetes/pull/717))
- validate mysql image and mysqlversion on cluster create ([radondb-mysql-kubernetes#715](https://github.com/radondb/radondb-mysql-kubernetes/pull/715))
- Support running single node.  ([radondb-mysql-kubernetes#584](https://github.com/radondb/radondb-mysql-kubernetes/pull/584))
- feat(e2e): add podchaos api and test cases ([radondb-mysql-kubernetes#697](https://github.com/radondb/radondb-mysql-kubernetes/pull/721))
- feat(backup)：Support fast backup ([radondb-mysql-kubernetes#693](https://github.com/radondb/radondb-mysql-kubernetes/pull/721))
- support MySQL configuration template ([radondb-mysql-kubernetes#656](https://github.com/radondb/radondb-mysql-kubernetes/pull/656))
- Support MySQL subversion ([radondb-mysql-kubernetes#647](https://github.com/radondb/radondb-mysql-kubernetes/pull/647))
- add disk usage check in nfs backup ([radondb-mysql-kubernetes#679](https://github.com/radondb/radondb-mysql-kubernetes/pull/679))
- api,backup: add printcolumn for Success ([radondb-mysql-kubernetes#670](https://github.com/radondb/radondb-mysql-kubernetes/pull/670))
- update go version to 1.17 ([radondb-mysql-kubernetes#677](https://github.com/radondb/radondb-mysql-kubernetes/pull/677))
- chart: Support the use of Helm to install MySQL cluster ([radondb-mysql-kubernetes#519](https://github.com/radondb/radondb-mysql-kubernetes/pull/519))

#### ECK:

- Implement a Hot-Warm-Cold deployment architecture
- Deploy Kibana to manage clusters in a visualized manner
- Integrated Elasticsearch Exporter with Prometheus

#### Kafka:

- Support Kafka 3.1 and 3.2
- Install and manage Kafka and ZooKeeper nodes on the GUI
- Support automatic cluster failover and recovery
- Dynamically expand the cluster capacity and storage
- Integrate with Prometheus to support the Exporter-based resource status monitoring feature
- Support TLS to enhance the security of internal access
- Deploy and manage users and topics and display the ACL on the GUI
- Support cluster rebalancing on the GUI

#### MongoDB:

- Add the service monitoring feature of MongoDB clusters
- Improve database account management
- Support dynamic disk expansion
- Optimize the elegant start and stop logic
- Display more cluster states

#### OpenSearch:

- Support installation and management of OpenSearch, Dashboard, and Logstash
- Support OpenSearch 2.3.0
- Dynamically increase the compute specifications and storage capacity of clusters
- Dynamically adjust the number of cluster nodes
- Perform a rolling update upon configurations change
- Support the keystore feature
- Integrate with Prometheus to support the Exporter-based resource status monitoring feature
- Support TLS

#### PostgreSQL:

- Support PostgreSQL kernel versions 12.12, 13.8, 14.5, and 15.0
- Support PostGIS and Citus plugins
- Support scale-in/scale-out and scale-up/scale-down
- Support semi-sync streaming replication
- Display extensive monitoring metrics




#### Redis:

- Build based on Redis 6.x and 7.x
- Introduce Proxy for clusters
- Support Redis Cluster failover and auto-recovery
- Suport resource status monitoring based on the Prometheus Exporter
- Support Kubernetes requests and limits
- Support password and password-free settings for clusters
- Support hot updates of Redis's management configuration parameters
 
## App Store

### Features

- Support to manage helm releases created by the native helm command ([kse#317](https://github.com/kubesphere/kse/pull/317) by [@xyz-li](https://github.com/xyz-li), [kse-console#192](https://github.com/kubesphere/kse-console/pull/192) by [@harrisonliu5](https://github.com/harrisonliu5))
- Add new options in openpitrix. ([kse-installer#62](https://github.com/kubesphere/kse-installer/pull/62), [@xyz-li](https://github.com/xyz-li))

### Bug Fixes
- Fix a ks-controller-manager crash error caused by Helm controller NPE errors. ([kubesphere#4602](https://github.com/kubesphere/kubesphere/pull/4602), [@chaunceyjiang](https://github.com/chaunceyjiang))
- Fix an issue when updating traffic policy in traffic monitor page of an application. ([console#3836](https://github.com/kubesphere/console/pull/3836), [@mujinhuakai](https://github.com/mujinhuakai))

## Network

### Enhancements & Updates

- Integrate OpenELB with KubeSphere for exposing LoadBalancer services. ([console#2993](https://github.com/kubesphere/console/pull/2993), [@weili520](https://github.com/weili520))

### Bug Fixes

- Fix an issue where the gateway of a project is not deleted after the project is deleted. ([kubesphere#4626](https://github.com/kubesphere/kubesphere/pull/4626), [@RolandMa1986](https://github.com/RolandMa1986))
- Fix an issue during gateway log query, where the log query connection is not terminated after the query is complete. ([kubesphere#4927](https://github.com/kubesphere/kubesphere/pull/4927), [@qingwave](https://github.com/qingwave))
- Fix an issue where users fail to create routing rules in IPv6 and IPv4 dual-stack environments.([console#3604](https://github.com/kubesphere/console/pull/3604), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao))
- Update the OpenELB check URL. ([console#3550](https://github.com/kubesphere/console/pull/3550), [@harrisonliu5](https://github.com/harrisonliu5))


## Service Mesh

### Bug Fixes

- Resolve port conflicts of virtual services that use multiple protocols. ([kubesphere#4560](https://github.com/kubesphere/kubesphere/pull/4560), [@RolandMa1986](https://github.com/RolandMa1986))

## KubeEdge Integration

### Features
- Add support for logging in to common cluster nodes and edge nodes from the KubeSphere web console. ([kubesphere#4579](https://github.com/kubesphere/kubesphere/pull/4579), [@lynxcat](https://github.com/lynxcat), [console#2888](https://github.com/kubesphere/console/pull/2888), [@lynxcat](https://github.com/lynxcat))

### Enhancements & Upgrades
- Upgrade KubeEdge from v1.7.2 to v1.9.2.
- Remove EdgeWatcher as KubeEdge v1.9.2 provides similar functions.

### Bug Fixes
- Change the cluster module key from `kubeedge` to `edgeruntime`.([console#3548](https://github.com/kubesphere/console/pull/3548), [@harrisonliu5](https://github.com/harrisonliu5))

## License

### Features

- Add Cluster ID to the license control mechanism ([kse#291](https://github.com/kubesphere/kse/pull/291) by [@xyz-li](https://github.com/xyz-li), [kse-console#183](https://github.com/kubesphere/kse-console/pull/183) by [@harrisonliu5](https://github.com/harrisonliu5))
- Add license for Observability Center ([kse-console#258](https://github.com/kubesphere/kse-console/pull/258), [@harrisonliu5](https://github.com/harrisonliu5))
- Add license for Spring Cloud.
- Add metrics and alert rules for the KubeSphere licence, notifications will be sent if the KubeSphere license is about to expire in less than 30 days ([kse#324](https://github.com/kubesphere/kse/pull/324) by [@xyz-li](https://github.com/xyz-li), [kse-installer#65](https://github.com/kubesphere/kse-installer/pull/65) by [@junotx](https://github.com/junotx))
- Support showing the License ID in the license page.

## Security

### Features

- Add Ingress for ks-console to support HTTPS ([kse#327](https://github.com/kubesphere/kse/pull/327), [@iawia002](https://github.com/iawia002))

### Enhancements & Updates

- Enforce the password length to be at least 8 characters
- Add the logic to check if the username and password are same 
- Disable password autofilling ([kse-console#190](https://github.com/kubesphere/kse-console/pull/190), [@harrisonliu5](https://github.com/harrisonliu5))

## Development & Testing

### Features

- Add the --controllers flag to ks-controller-manager, which allows users to select controllers to be enabled for resource usage reduction during debugging. ([kubesphere#4512](https://github.com/kubesphere/kubesphere/pull/4512), [@live77](https://github.com/live77))
- Prevent ks-apiserver and ks-controller-manager from restarting when the cluster configuration is changed. ([kubesphere#4659](https://github.com/kubesphere/kubesphere/pull/4659), [@x893675](https://github.com/x893675))
- Add an agent to report additional information about ks-apiserver and controller-manager in debugging mode. ([kubesphere#4928](https://github.com/kubesphere/kubesphere/pull/4928), [@xyz-li](https://github.com/xyz-li))

## User Experience

### Features

- Now events can be displayed for abnormal pods ([kse-console#196](https://github.com/kubesphere/kse-console/pull/196), [@yazhouio](https://github.com/yazhouio))
- Add in-memory cache in addition to the original Redis cache ([kse#359](https://github.com/kubesphere/kse/pull/359), [@zhou1203](https://github.com/zhou1203))
- Update product name and logo to KubeSphere Enterprise ([kse-console#208](https://github.com/kubesphere/kse-console/pull/208), [@harrisonliu5](https://github.com/harrisonliu5))
- Add the End User License Agreements.
- Add EULA in password confirm ([kse-console#181](https://github.com/kubesphere/kse-console/pull/181), [@harrisonliu5](https://github.com/harrisonliu5))
- Add support for more languages on the KubeSphere web console. ([console#2782](https://github.com/kubesphere/console/pull/2782), [@xuliwenwenwen](https://github.com/xuliwenwenwen))
- Add the lifecycle management feature for containers. ([console#2940](https://github.com/kubesphere/console/pull/2940), [@harrisonliu5](https://github.com/harrisonliu5))
- Add support for creating container environment variables in batches from secrets and configmaps. ([console#3044](https://github.com/kubesphere/console/pull/3044), [@weili520](https://github.com/weili520))
- Add a message in the **Audit Log Search** dialog box, which prompts users to enable the audit logs feature. ([console#3062](https://github.com/kubesphere/console/pull/3062), [@harrisonliu5](https://github.com/harrisonliu5))
- Add data units in the **Create Storage Class** dialog box. ([console#3200](https://github.com/kubesphere/console/pull/3200), [@weili520](https://github.com/weili520))
- Add the cluster viewing permission to a user when the user adds a cluster. ([console#3296](https://github.com/kubesphere/console/pull/3296), [@harrisonliu5](https://github.com/harrisonliu5))
- Add a time range selector on the **Traffic Monitoring** tab page. ([console#2916]( https://github.com/kubesphere/console/pull/2916), [@weili520](https://github.com/weili520))
- Add a prompt to remind users to select a language or artifact type when users are building images. ([console#3534](https://github.com/kubesphere/console/pull/3534), [@harrisonliu5](https://github.com/harrisonliu5))
- Add support for changing the number of items displayed on each page of a table.([console#3486](https://github.com/kubesphere/console/pull/3486), [@weili520](https://github.com/weili520))
- Add support for batch stopping workloads. ([console#3497](https://github.com/kubesphere/console/pull/3497), [@harrisonliu5](https://github.com/harrisonliu5))
- Add the creator annotation to ensure information displayed on the pod detail page is consistent with other details pages. ([console#3820](https://github.com/kubesphere/console/pull/3820), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao))
- Add support for displaying the revision record when the workload type is `statefulsets` or `daemonsets`.([console#3819](https://github.com/kubesphere/console/pull/3819), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao))
- Add a support page. ([console#3832](https://github.com/kubesphere/console/pull/3832), [@yazhouio](https://github.com/yazhouio))
- Add support for duplicate name validation of containers.  ([console#3559](https://github.com/kubesphere/console/pull/3559), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao))
- Add support for duplicate name validation of service names. ([console#3696](https://github.com/kubesphere/console/pull/3696), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao))

### Enhancements & Updates

- Support setting the default container registry ([kse-console#197](https://github.com/kubesphere/kse-console/pull/197), [@yazhouio](https://github.com/yazhouio))
- Support the shortcut dropdown of the relevent events of the pods.
- Support the quick jumping to the destination page and setting the rows quantity showed of the page.
- Support to hide external links in the console
- Remove the limit to 100 of the cpu-percent of HPA.
- Optimize the service details area on the **Service Topology** page. ([console#2945](https://github.com/kubesphere/console/pull/2945), [@tracer1023](https://github.com/tracer1023))
- Prevent passwords without uppercase letters set through the backend CLI. ([kubesphere#4481](https://github.com/kubesphere/kubesphere/pull/4481), [@live77](https://github.com/live77))
- Optimize the UI text of cluster, workspace, and project deletion. ([console#3004](https://github.com/kubesphere/console/pull/3004), [@weili520](https://github.com/weili520))
- Set the **Token** parameter on the webhook settings page as mandatory. ([console#2903](https://github.com/kubesphere/console/pull/2903), [@xuliwenwenwen](https://github.com/xuliwenwenwen))
- Improve user experience on the web console of member clusters. ([kubesphere#4721](https://github.com/kubesphere/kubesphere/pull/4721), [@iawia002](https://github.com/iawia002), [console#3031](https://github.com/kubesphere/console/pull/3031), [@weili520](https://github.com/weili520))
- Optimize the **Service Type** and **External Access** columns of the service list. ([console#3058](https://github.com/kubesphere/console/pull/3058),[@weili520](https://github.com/weili520))
- Change the abbreviation of "ReadOnlyMany" from "ROM" to "ROX". ([console#2751](https://github.com/kubesphere/console/pull/2751), [@123liubao](https://github.com/123liubao))
- Improve the registry verification API to resolve registry verification failures. ([kubesphere#4678](https://github.com/kubesphere/kubesphere/pull/4678), [@wansir](https://github.com/wansir))
- Optimize image building of KubeSphere and the console.([console#3610](https://github.com/kubesphere/console/pull/3610), [@zt1046656665](https://github.com/zt1046656665))

### Bug Fixes
- Fix an issue where statefulset creation fails when a volume is mounted to an init container. ([console#2730](https://github.com/kubesphere/console/pull/2730), [@weili520](https://github.com/weili520))
- Fix an app installation failure, which occurs when users click buttons too fast. ([console#2735](https://github.com/kubesphere/console/pull/2735), [@weili520](https://github.com/weili520))
- Fix an incorrect message displayed during workload creation, which indicates that resource requests have exceeded limits. ([console#2809](https://github.com/kubesphere/console/pull/2809) , [@weili520](https://github.com/weili520))
- Fix an issue where no data is displayed on the **Traffic Management** and **Tracing** tab pages in a multi-cluster project. ([console#3195](https://github.com/kubesphere/console/pull/3195), [@harrisonliu5](https://github.com/harrisonliu5))
- Fix an issue where data in the service details area on the **Service Topology** page is not updated automatically. ([console#3117](https://github.com/kubesphere/console/pull/3117),[@weili520](https://github.com/weili520))
- Fix an issue where existing settings are not displayed when a stateful service is edited. ([console#2845](https://github.com/kubesphere/console/pull/2845), [@weili520](https://github.com/weili520))
- Fix an issue where container probes are still displayed after they are deleted. ([console#3213](https://github.com/kubesphere/console/pull/3213), [@harrisonliu5](https://github.com/harrisonliu5))
- Fix incorrect container names in the service details area on the **Service Topology** page. ([console#3276](https://github.com/kubesphere/console/pull/3276), [@weili520](https://github.com/weili520))
- Fix the incorrect number of worker nodes on the **Cluster Nodes** page. ([console#3279](https://github.com/kubesphere/console/pull/3279), [@weili520](https://github.com/weili520))
- Fix a workspace API error caused by an incorrect cluster role. ([console#3297](https://github.com/kubesphere/console/pull/3297), [@harrisonliu5](https://github.com/harrisonliu5))
- Fix an incorrect error message displayed when kubeconfig is updated for a second time. ([console#3392](https://github.com/kubesphere/console/pull/3392), [@harrisonliu5](https://github.com/harrisonliu5))
- Fix goroutine leaks that occur when a container terminal is opened. ([kubesphere#4918](https://github.com/kubesphere/kubesphere/pull/4918), [@anhoder](https://github.com/anhoder))
- Fix unexpected ks-apiserver panic caused by an index out of range error of metrics API calling. ([kubesphere#4691](https://github.com/kubesphere/kubesphere/pull/4691), [@larryliuqing](https://github.com/larryliuqing))
- Fix an ks-apiserver crash caused by resource discovery failure. ([kubesphere#4835](https://github.com/kubesphere/kubesphere/pull/4835),  [@wansir](https://github.com/wansir))
- Fix an incorrect private key type in kubeconfig information. ([kubesphere#4936](https://github.com/kubesphere/kubesphere/pull/4936), [@xyz-li](https://github.com/xyz-li))
- Fix an issue where the key is not display in LoadBalancer. ([console#3503](https://github.com/kubesphere/console/pull/3503), [@harrisonliu5](https://github.com/harrisonliu5))
- Fix an issue where no prompt is displayed when users do not fill in key and value in the LoadBalancer access mode.([console#3499](https://github.com/kubesphere/console/pull/3499), [@weili520](https://github.com/weili520))
- Fix an issue where the update time of a service is incorrect on the detail page.([console#3803](https://github.com/kubesphere/console/pull/3803), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao))
- Fix inaccurate prompt when users are adding an init container.([console#3561](https://github.com/kubesphere/console/pull/3561), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao))
- Fix an issue where an error occurs while users enter Chinese characters in secret. ([console#3774](https://github.com/kubesphere/console/pull/3774), [@moweiwei](https://github.com/moweiwei))
- Fix an issue where the total number of pages is incorrectly displayed.([kubesphere#5201](https://github.com/kubesphere/kubesphere/pull/5201), [@yongxingMa](https://github.com/yongxingMa))
- Fix an issue where the update time of an application is incorrect. ([console#3541](https://github.com/kubesphere/console/pull/3541), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao))
- Fix a 404 issue in Documentation. ([console#3484](https://github.com/kubesphere/console/pull/3433) [@PrajwalBorkar](https://github.com/PrajwalBorkar))
- Fix an issue where the status of a cluster remains true when the cluster fails to join Federation. ([kubesphere#5137](https://github.com/kubesphere/kubesphere/pull/5137), [@x893675](https://github.com/x893675))
- Fix an issue where traffic allocation fails in the canary release mode. ([console#3542](https://github.com/kubesphere/console/pull/3542), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao))
- Fix an issue where configurations do not take effect when users set the pod request to 0.([console#3827](https://github.com/kubesphere/console/pull/3827), [@harrisonliu5](https://github.com/harrisonliu5))
- Fix an issue where canary release goes wrong when multiple operating systems are selected.([console#3479](https://github.com/kubesphere/console/pull/3479), [@zhaohuiweixiao](https://github.com/zhaohuiweixiao))
- Fix an issue where configmap configurations cannot be saved while users are creating a workload.([console#3416](https://github.com/kubesphere/console/pull/3416),[@weili520](https://github.com/weili520))

## API Changes

- Alerting API has been upgraded to v2beta1, and the v2alpha1 alerting API is deprecated ([kse#331](https://github.com/kubesphere/kse/pull/331), [@junotx](https://github.com/junotx))
- Expose kube-apiserver of the host cluster as a LoadBalancer service for member clusters to access. ([kubesphere#4528](https://github.com/kubesphere/kubesphere/pull/4528), [@lxm](https://github.com/lxm))
- Provide RESTful APIs for ClusterTemplate. ([ks-devops#468](https://github.com/kubesphere/ks-devops/pull/468), [@JohnNiang](https://github.com/JohnNiang))
- Provide template-related APIs. ([ks-devops#460](https://github.com/kubesphere/ks-devops/pull/460), [@JohnNiang](https://github.com/JohnNiang))
- Change the KubeEdge proxy service to `http://edgeservice.kubeedge.svc/api/`. ([kubesphere#4478](https://github.com/kubesphere/kubesphere/pull/4478), [@zhu733756](https://github.com/zhu733756))
- Change the patch type of `PatchWorkspaceTemplate` from `MergePatchType` to `JSONPatchType`.([kubesphere#5217](https://github.com/kubesphere/kubesphere/pull/5217), [@zhou1203](https://github.com/zhou1203))