# Self-assessment for Antrea

December 2023

*_Authors_*: Skye Kim, Vamsi Koneti, Ethan Shieh, Chris Torres

## Table of Contents

* [Metadata](#metadata)
* [Security links](#security-links)
* [Overview](#overview)
* [Actors](#actors)
* [Actions](#actions)
* [Background](#background)
* [Goals](#goals)
* [Non-goals](#non-goals)
* [Self-assessment use](#self-assessment-use)
* [Security functions and features](#security-functions-and-features)
* [Project compliance](#project-compliance)
* [Secure development practices](#secure-development-practices)
* [Security issue resolution](#security-issue-resolution)
* [Appendix](#appendix)

## Metadata

| | |
| -- | -- |
| Software | https://github.com/antrea-io/antrea |
| Security Provider | No |
| Languages | Go, Shell, PowerShell, Makefile, Dockerfile, Jinja |
| SBOM | https://app.fossa.com/projects/git%2Bgithub.com%2Fantrea-io%2Fantrea |
| | |

### Security links

| Doc | url |
| -- | -- |
| Security file | https://github.com/antrea-io/antrea/blob/main/docs/security.md |
| Default and optional configs | https://github.com/antrea-io/antrea/blob/main/docs/configuration.md |

## Overview

Antrea is a Kubernetes network plugin that uses Open vSwitch to extend the
capabilities and observability of a Kubernetes cluster's network.

### Background

Antrea is a Kubernetes-native project that utilizes the Container Network Interface (CIN) and Kubernetes NetworkPolicy. The tool facilitates communication between the containers in a Kubernetes cluster, handling aspects like network routing and security policies. Networking in Kubernetes can be complex due to the dynamic nature of container deployment and scaling, and Antrea aims to simplify and secure this process. By leveraging Open vSwitch (OVS), Antrea extends programmable networks to Kubernetes, providing a unified network stack that simplifies networking across diverse clouds and operating systems. Antrea is designed to make it easier for users to troubleshoot through Kubernetes controller patterns and diagnostics compatible with familiar network operator tools. Open vSwitch also allows for advanced network use cases to be implemented, such as kernel bypass and network service mesh.

### Actors
The Antrea Agent: A primary actor in this ecosystem is the Antrea Agent, deployed on each Kubernetes node. This agent is pivotal in managing the networking of pods, interfacing directly with the Open vSwitch (OVS) to execute data plane operations. It applies the network policies as defined in Kubernetes, thereby playing a critical role in regulating network traffic and enforcing security rules at the node level.

The Antrea Controller: Complementing the Antrea Agent is the Antrea Controller, a centralized component that administers the entire Antrea CNI deployment across the cluster. Its primary function is to process network policies, ensuring uniform implementation throughout the Kubernetes cluster. The Antrea Controller maintains an overarching view of the network state and policies, facilitating a coordinated approach to network management and security.

Kubernetes API Server Interaction: External to Antrea but integral to its operation is the Kubernetes API Server. This actor serves as the communication hub for Antrea, providing the necessary information about network policies, pod lifecycle events, and other essential Kubernetes resources. The interaction between Antrea and the Kubernetes API Server is fundamental for the dynamic adaptation of network configurations in response to changes within the Kubernetes environment.

Open vSwitch (OVS): At the heart of the data plane operations lies the Open vSwitch (OVS), a high-performance, multilayer virtual switch. OVS is responsible for forwarding network packets based on the configurations and policies set by the Antrea Agents. It forms the backbone of the network’s data plane, efficiently handling network traffic within the cluster.

### Actions
In a Kubernetes cluster, Antrea initializes the Antrea Agent and the Antrea Controller. The Antrea Agent sets up Open vSwitch (OVS) bridges on each Kubernetes Node to create connections between Pods and manages the IP address allocation. The Antrea Controller watches for changes in the NetworkPolicy, Pod, and Namespace resources from the Kubernetes API. Based on instructions from the Antrea Controller, the Antrea Agent enforces network policies locally by creating OVS flows on each Node.

### Goals
The main goal of Antrea is to enhance network connectivity and security within Kubernetes clusters. It focuses on ensuring that containers can communicate efficiently and securely, using strict network policies to allow only authorized traffic. Antrea uses Open vSwitch for high-performance networking, making it suitable for large deployments. In terms of security, Antrea encrypts data moving across the cluster to protect against potential breaches. However, it's important to note that Antrea primarily addresses network-level security, not broader security issues like application vulnerabilities. Overall, Antrea's aim is to provide a reliable and secure networking solution for Kubernetes environments.

### Non-goals
Antrea is designed specifically for Kubernetes clusters and primarily focuses on managing internal communication within the clusters rather than interaction with external devices. It is not suitable for non-Kubernetes environments. Antrea isn't designed to be an all-in-one security solution for Kubernetes clusters. It focuses on network communications but does not provide any application layer services or protection against application-level vulnerabilities. Antrea is also not a full-fledged network monitoring tool. While it offers some basic insights into network traffic, it doesn't provide deep network analysis or troubleshooting features. Lastly, Antrea doesn't control how much network resources (like bandwidth) each container or service uses in the cluster.

## Self-assessment use

This self-assessment is created by the Antrea team to perform an internal analysis of the
project's security. It is not intended to provide a security audit of Antrea, or
function as an independent assessment or attestation of Antrea's security health.

This document serves to provide Antrea users with an initial understanding of
Antrea's security, where to find existing security documentation, Antrea plans for
security, and general overview of Antrea security practices, both for development of
Antrea as well as security of Antrea.

This document provides the CNCF TAG-Security with an initial understanding of Antrea
to assist in a joint-assessment, necessary for projects under incubation. Taken
together, this document and the joint-assessment serve as a cornerstone for if and when
Antrea seeks graduation and is preparing for a security audit.

## Security functions and features
Antrea, as a Kubernetes-native networking and security solution, incorporates several features and practices to ensure security within a Kubernetes cluster. Here's a security overview of Antrea:
1. **[Network Policies](https://antrea.io/docs/v1.14.0/):**
Antrea provides a comprehensive network policy model, building upon Kubernetes Network Policies. It includes features such as policy tiering, rule priorities, and cluster-level policies.
Network Policies define and enforce rules for controlling the traffic between Pods, contributing to the segmentation and isolation of applications.
2. **Centralized Policy Computation:**
Antrea's design involves
[centralized policy computation](https://antrea.io/docs/v1.3.0/docs/design/architecture/)
by the Antrea Controller. This approach ensures consistency across Nodes, simplifies management, and aids in debugging.
Only one Antrea Controller instance processes NetworkPolicy updates, reducing the overall cost compared to performing complex policy computation on every Node.
3. **Encrypted Pod Traffic:**
Antrea supports encrypting inter-Node Pod traffic using IPsec or WireGuard tunnels, enhancing the confidentiality and integrity of communication between Pods.
[Traffic encryption](https://antrea.io/docs/v1.11.4/docs/traffic-encryption/),
however, is not supported on Windows Nodes yet
4. **Security Policies for VMs:**
Antrea
[native policies](https://antrea.io/docs/v1.14.0/)
can be enforced not only on Kubernetes Nodes but also on non-Kubernetes Nodes, including virtual machines (VMs) and bare-metal servers.
5. **Authentication and Authorization:**
Antrea Controller leverages Kubernetes
[ServiceAccount](https://antrea.io/docs/v1.3.0/docs/design/architecture/)
tokens for authentication. Antrea Agents use a Kubernetes ServiceAccount token to authenticate with the Controller.
Authentication and authorization are delegated to the Kubernetes API, enhancing the overall security of communication within the cluster.
6. **gRPC and Protobuf:**
Antrea Agent exposes a
[gRPC](https://antrea.io/docs/v1.2.0/docs/design/architecture/)
service (CNI service) for communication with the antrea-cni binary. gRPC communication ensures secure and efficient communication between components.
Messages between the Controller and Agent are serialized using the Protobuf format, reducing message size and enhancing efficiency.
7. **Security Monitoring and Diagnostics:**
Antrea comes with CLI and UI tools that provide visibility and diagnostic capabilities. These tools include
[packet tracing, policy analysis,](https://github.com/antrea-io/antrea/blob/main/README.md)
and flow inspection, aiding in troubleshooting.
The exposure of Prometheus metrics facilitates security monitoring and enables the integration of Antrea with monitoring solutions.
8. **Security for Windows Nodes:**
Antrea is designed to be compatible with both Linux and Windows Kubernetes Nodes. This enables a consistent security posture across heterogeneous environments.
9. **ClusterIP Service Security:**
Antrea supports both
[kube-proxy](https://antrea.io/docs/main/docs/design/architecture/)
and AntreaProxy for ClusterIP Service load balancing. AntreaProxy implementation within OVS enhances security by minimizing overhead associated with forwarding Service traffic to the host's network stack.
10. **Antrea Multi-cluster Security:**
Antrea Multi-cluster implements a
[Multi-cluster Service API](https://antrea.io/docs/main/docs/multicluster/architecture/),
allowing for the creation of secure multi-cluster Services. It supports ClusterNetworkPolicy replication across clusters, enhancing security across a ClusterSet.
11. **IPsec Encryption:**
Antrea supports
[IPsec](https://antrea.io/docs/v1.11.4/docs/traffic-encryption/)
for encrypting Pod traffic across Linux Nodes. This adds an additional layer of security to communication within the cluster.
12. **Network Flow Visibility:**
Antrea supports exporting network flow information using
[IPFIX](https://antrea.io/docs/v1.2.3/docs/getting-started/),
which can be visualized with Elastic Stack and Kibana dashboards. This visibility enhances security monitoring capabilities.
13. **Web UI Security:**
The Antrea web UI provides a graphical interface for monitoring. Access to this UI and the information it displays is controlled by
[Kubernetes RBAC](https://antrea.io/docs/v1.3.0/docs/antrea-network-policy/),
ensuring secure access control.

## Project compliance

Regarding the compliance of the Antrea project with established security standards, there is no direct documentation or claim that Antrea meets specific standards like PCI-DSS, COBIT, ISO standards, GDPR, or others. Antrea's primary role is as a network plugin for Kubernetes, focusing on network connectivity and security within Kubernetes clusters. Although it incorporates security measures that may align with aspects of these standards, particularly in network security and data protection, it does not explicitly conform to these standards as a standalone entity. For organizations using Antrea, ensuring compliance with such standards would typically involve evaluating their entire Kubernetes architecture, including Antrea, as part of a comprehensive compliance strategy. Thus, while Antrea plays a role in creating a secure network environment, its compliance with specific security standards would ultimately be determined by how it is integrated and used within an organization's broader Kubernetes infrastructure.

## Secure development practices

### Development Pipeline

Development on Antrea is done on GitHub. To ensure that contributions are
safe and properly reviewed prior to being added to the main branch of the
Antrea repository, the Antrea team follows the secure development practices of:

* Installing
[Git client-side hooks](https://github.com/antrea-io/antrea/blob/main/hack/git_client_side_hooks)
to:
* Run golangci-lint to check source code for style and safety before a commit.
* Run scripts to check
[documentation spelling](https://github.com/antrea-io/antrea/blob/main/hack/verify-spelling.sh),
[formatting](https://github.com/antrea-io/antrea/blob/main/hack/verify-docs-for-website.sh),
and to update
[table of contents](https://github.com/antrea-io/antrea/blob/main/hack/update-toc.sh)
before a commit.
* Check that a
[Developer Certificate of Origin](https://developercertificate.org/)
(DCO) is present for every new commit before a push.
* Making changes in a new branch of a fork of the Antrea repository and
submitting a pull request (PR) before their change can be integrated into
the main repository.
* Having multiple reviewers check over a PR and provide feedback before the
change is merged by a maintainer. Minor changes can be reviewed and approved
by one team member, but most PRs are reviewed by 2-3 team members.
* Running
[GitHub Action and Jenkins continuous integration (CI)](https://github.com/antrea-io/antrea/blob/main/ci/README.md)
checks relevant
to their contribution when a PR is made. These include
[Go linters](https://github.com/antrea-io/antrea/blob/main/ci/README.md#go-linters),
unit tests (that use the
[Go testing](https://golang.org/pkg/testing/)
package),
[integration tests](https://github.com/antrea-io/antrea/blob/main/test/integration),
[end-to-end tests](https://github.com/antrea-io/antrea/blob/main/test/e2e), and
[Kubernetes upstream tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md).
* Daily and weekly Jenkins tests are automatically run on the main branch of
the Antrea repository. Developers are informed of test failures through
the
[developer mailing list](https://groups.google.com/forum/#!forum/projectantrea-dev).
* Using a
[DCO GitHub App](https://github.com/apps/dco)
to check that a DCO is present in every commit of a PR before review of that
PR begins.
* Cherry-picking PRs with bugfixes to older, currently-supported releases so
that every currently-supported version of Antrea benefits from the
contribution.

### Communication Channels

The Antrea project uses a variety of communication channels for internal,
inbound, and outbound communications. This section lists the communication
channels that are publicly listed on the GitHub project README.

* Internal:
* Antrea team members communicate through the
[Antrea channel](https://kubernetes.slack.com/messages/CR2J23M0X)
on the Kubernetes Slack. The Slack channel is used by the maintainers to
post announcements and discuss PRs.
* The Antrea GitHub project's
[issues page](https://github.com/antrea-io/antrea/issues)
to track feature requests, known bugs, and proposals. The team also uses
the GitHub project's
[PR page](https://github.com/antrea-io/antrea/pulls)
to manage and review contributions to the project.
* Synchronous community meetings are held biweekly on Tuesdays for team
members to discuss releases, feature proposals, and user issues. Feature
proposals are typically presented in slideshow form. The
[minutes](https://github.com/antrea-io/antrea/wiki/Community-Meetings)
and
[recording](https://www.youtube.com/playlist?list=PLuzde2hYeDBdw0BuQCYbYqxzoJYY1hfwv)
for each meeting are openly available.
* The Antrea team uses a
[developer mailing list](https://groups.google.com/forum/#!forum/projectantrea-dev)
to keep the team updated on daily and weekly Jenkins build failures. The
mailing list was briefly formerly used for internal development communication.

* Inbound
* [Live office hours](https://calendar.google.com/calendar/u/0/embed?src=[e-mail])
are held biweekly on Tuesdays when community meetings are
not held. During office hours, users can join and ask Antrea team members
questions.
* Users can contact the Antrea team through the
[Antrea channel](https://kubernetes.slack.com/messages/CR2J23M0X)
on the Kubernetes slack for questions and support.
* Users can post bug reports, feature requests, and support requests on the
[issues page](https://github.com/antrea-io/antrea/issues)
of the project's GitHub repository.
* Users can report security vulnerabilities by emailing the Antrea maintainers
at the address [e-mail].

* Outbound
* The Antrea team occasionally uses their live office hours time to hold a
[Antrea LIVE](https://www.youtube.com/@projectantrea/streams),
a livestream about Antrea (releases and usage) and topics related to Antrea
such as Open vSwitch, CNI (Container Network Inferface), and networking
in Kubernetes.
* The Antrea team formerly used a
[announcement mailing list](https://groups.google.com/g/projectantrea)
to send project announcements to the community. The team also formerly used a
[general mailing list](https://groups.google.com/g/projectantrea) to send
community meeting reminders and agendas. Usage of this channel was
discontinued around September 2022. The team now uses
[Slack](https://kubernetes.slack.com/messages/CR2J23M0X)
for these two types of communications.
* The Antrea team formerly used
[X/Twitter](https://twitter.com/ProjectAntrea)
to announce Antrea LIVE livestreams and major releases. Usage of this
channel was discontinued around November 2022.

### Ecosystem

Antrea is a Kubernetes-native network plugin that implements networking using
Kubernetes' [Container Network Interface](https://github.com/containernetworking/cni)
(CNI) to manage pod network interfaces. The project is designed primarily for
use in Kubernetes clusters.

Antrea works to expand the cloud native ecosystem by extending the networking
capabilities of Kubernetes. The project adds more network policies, support for
multi-OS clusters (Linux and Windows), plugin-native troubleshooting tools
(traceflows), offloading networking to hardware on a node, and encrypted
inter-node networking (IPsec or WireGuard).

Other Kubernetes CNI plugins exist, such as
[Calico](https://github.com/projectcalico/calico),
[Flannel](https://github.com/flannel-io/flannel),
and [Cilium](https://github.com/cilium/cilium). The main benefit to the
cloud-native ecosystem that Antrea provides is an implementation of the CNI
using
[Open vSwitch](https://www.openvswitch.org/).
Open vSwitch is a virtual switch that operates in the
data plane that creates virtual interfaces for pods and tunnels between nodes.
By using Open vSwitch, Antrea can support
[much higher network policy performance, processor utilization, and packet throughput](https://www.vmware.com/products/antrea-container-networking.html)
than CNI plugins that use more common data plane technology like
[eBPF](https://ebpf.io/),
[built-in Linux bridges](https://wiki.linuxfoundation.org/networking/bridge),
or
[Host Network Service](https://learn.microsoft.com/en-us/virtualization/windowscontainers/container-networking/architecture)
(for Windows). Another benefit of Open vSwitch is its Windows support --
this reduces the complexity of development and usage because only one data
plane technology needs to be supported.

For companies that want to run applications on a cloud environment,
[VMware offers enterprise-grade technical support](https://www.vmware.com/products/antrea-container-networking.html)
for companies using Antrea alongside their cloud application on VMware Tanzu
or vSphere. In addition, VMware sponsors development on the Antrea project so
that there is always active work being done on the project. All of the
maintainers and some of the contributors to the project are affiliated with
VMware.

## Security issue resolution

### Responsible Disclosures Process

If a security issue is found in Antrea, the discoverer of that issue is
instructed to report the issue directly to the maintainers at the email address
[e-mail]. Any kind of security vulnerability,
irregardless of severity, should be reported to the maintainers so that the
issue is not made public before a fix is developed. When reporting the
vulnerability to the maintainers, the reporter must describe how to reproduce the
issue (including the software that they used), what the reporter believes the
attack vector and attack surface to be, the impact of the vulnerability on
a cluster if exploited, and the impact of the vulnerability on Antrea components.

Once a report is received by the maintainers, a maintainer (the coordinator of
that issue) is responsible with working with the reporter
to determine the severity of the vulnerability and to patch the vulnerability.
Vulnerabilities may be demoted to regular issues by the coordinator if the
maintainers are low risk.

The responsible disclosures process for Antrea is similar to the responsible
disclosures process of other large open source projects, like the
[Linux kernel](https://www.kernel.org/doc/html/latest/process/security-bugs.html),
[Kubernetes](https://kubernetes.io/docs/reference/issues-security/security/), and
[Cilium](https://github.com/cilium/cilium/blob/main/SECURITY.md)
in that vulnerabilities are privately reported to the maintainers. So, the
reporting process is satisfactory for ensuring that vulnerabilities are
responsibly disclosed at the appropriate time. To incentivize contributors to
find and fix vulnerabilities, the Antrea project can consider implementing a
bug bounty program in the future as
[Kubernetes](https://kubernetes.io/blog/2020/01/14/kubernetes-bug-bounty-announcement/)
has done. However, this is likely presently infeasible due to the small size of
the project.

### Incident Response

Security issues are triaged based on the risk of the issue. Issues with high
risk have a known, practical attack vector and can compromise the functionality
of Antrea or a Kubernetes cluster through Antrea. The Antrea maintainers use
a
[1-7 scale](https://github.com/antrea-io/antrea/blob/main/SECURITY.md#reference-taxonomy-for-issue-risk)
to score the risk of a vulnerability -- a 1 is given to the highest
risk vulnerabilities and a 7 is given to low risk ones. A score of 1-3 is given
to vulnerabilities that can be successfully exploited by a malicious actor if
known -- these are confirmed vulnerabilities. These vulnerabilities are fixed
privately by the coordinator (maintainer
assigned to handle the vulnerability) and the reporter. Vulnerabilities with a
score of 4-7 are issues that cannot be practically exploited, not a
vulnerability, or a vulnerability in an unsupported branch. These lower-risk
vulnerabilities will be demoted to a GitHub issue (become public) and handled as
an issue.

Once a patch for the vulnerability has been developed by the reporter (possibly
alongside the coordinator and/or other maintainers), the reporter emails the
patch to the address [e-mail] with a Git patch
containing the fix (patched from the current main branch). The maintainers will
then review the patch in the same way that a member of the Antrea team would
review a PR. The patch is sent via email rather than a GitHub PR to ensure that
the vulnerability remains confidential during the review process.

When the patch for a vulnerability is approved by the maintainers for merging,
the coordinator will create a GitHub issue for the vulnerability. At this point,
the vulnerability is publicly disclosed. Then, the coordinator submits PR with the
patch and a maintainer promptly merges the PR to the main branch. The PR can be
merged immediately because it had already been privately reviewed. The patch can
also be
[cherry-picked to other supported release branches](https://github.com/antrea-io/antrea/blob/main/docs/contributors/cherry-picks.md)
if necessary. This procedure ensures that the vulnerability is disclosed only
after the incident has been resolved to reduce the risk of the vulnerability
being realized in a real-world environment.

Though the coordinator typically discloses the vulnerability and submits the PR
with the patch, the original reporter is credited in the GitHub issue, PR, and
listed as the patch's commit author.

## Appendix

### Known Issues Over Time:

#### CoreOS

Antrea [issue #626](https://github.com/antrea-io/antrea/issues/626)
highlights a challenge with CoreOS Container Linux, which
reached its end-of-life on May 26, 2020, and no longer receives updates.
Users are advised to migrate to another operating system promptly. The
problem arises from the use of networkd for network configuration in
CoreOS, where the default configuration causes potential issues with
Antrea, particularly the gateway interface (antrea-gw0). If networkd is
restarted, it can lead to the loss of the interface's IP configuration
and the deletion of associated routes. To mitigate this, users are
recommended to create specific configuration files in the /etc/systemd/network/
directory, ensuring the Unmanaged option is set to 'yes' for the
relevant interfaces. Importantly, this fix is applicable to CoreOS
versions equal to or greater than 1262.0.0 (December 2016), as the
Unmanaged option was not supported in earlier versions.

#### Pod Traffic Shaping

Antrea provides support for Pod Traffic Shaping by leveraging the open-source bandwidth plugin maintained by the CNI project. This plugin requires the following Kernel modules: ifb, sch_tbf and sch_ingress. It seems that at the moment Photon OS 3.0 is built without the ifb Kernel module, which you can confirm by running modprobe --dry-run ifb: an error would indicate that the module is indeed missing. Without this module, Pods with the kubernetes.io/egress-bandwidth annotation cannot be created successfully. Pods with no traffic shaping annotation, or which only use the kubernetes.io/ingress-bandwidth annotation, can still be created successfully as they do not require the creation of an ifb device.

If Photon OS is patched to enable ifb, we will update this documentation to reflect this change, and include information about which Photon OS version can support egress traffic shaping.

#### Photon OS 3.0

Antrea issue [#591](https://github.com/antrea-io/antrea/issues/591) and [#1516](https://github.com/antrea-io/antrea/issues/1516) is as follows: if running Photon OS 3.0 on Kubernetes Nodes, users may encounter error
messages in antrea-agent logs indicating failed flow entry additions to
the OVS bridge. This issue arises if the Kernel lacks the
CONFIG_NF_CONNTRACK_ZONES option, recently enabled in Photon OS. To
confirm, users can check with the command grep CONFIG_NF_CONNTRACK_ZONES= /boot/config-uname -r``; absence of CONFIG_NF_CONNTRACK_ZONES=y signifies the issue. Resolution involves upgrading to Photon OS version >= 4.19.87-4 (Jan 2020) using tdnf upgrade linux-esx
on all Nodes. After the fix, Antrea Agents should function correctly.
For potential connectivity issues post-fix, adjusting Photon OS's
default strict firewall rules, particularly accepting traffic on the
Antrea-created gateway interface (antrea-gw0), is advised. Additionally,
Antrea supports Pod Traffic Shaping using the CNI bandwidth plugin, but
on Photon OS 3.0, the required ifb Kernel module might be absent,
impacting Pods with specific annotations. Photon OS patching to enable
ifb could address this limitation in future updates.

### OpenSSF Best Practices:
The Antrea project has achieved the passing level criteria and is in the process of working towards attaining a silver badge in Open Source Security Foundation (OpenSSF) best practices badge. [Antrea's OpenSSF Best Practices badge](https://www.bestpractices.dev/en/projects/4173).

### Antrea Adopters:
Here are a [few companies that have publicly shared the details of how they use Antrea](https://antrea.io/docs/main/adopters/):
1. Glasnostic: Glasnostic is a company that enhances the resilience of contemporary cloud operations by dynamically shaping the interactions among systems in real-time. This approach enables DevOps and SRE teams to achieve dependable deployments, proactively avoid failures, and ensure a consistent customer experience. Our utilization of Antrea's Open vSwitch support allows us to fine-tune the interactions between services within Kubernetes clusters.

2. Transwarp: Transwarp is an organization that is dedicated to advancing enterprise-level big data infrastructure software, providing comprehensive support across the entire data lifecycle to shape a future-oriented data world. In pursuit of robust security and seamless network operations within their Kubernetes platform, Transwarp relies on Antrea's AntreaClusterNetworkPolicy and AntreaNetworkPolicy to safeguard big data software for every tenant. The integration of Antrea's Open vSwitch is instrumental in establishing Pod-To-Pod networks, serving the needs of connectivity between flannel and antrea clusters, within antrea clusters, and facilitating smooth network transitions during upgrades within a single cluster. Furthermore, the utilization of Antrea's Egress feature ensures the retention of original source IPs, guaranteeing accurate identification of the real source IP for requests by Internal Pods. This strategic use of Antrea's features underscores Transwarp's commitment to delivering secure, efficient, and reliable big data infrastructure solutions for enterprises.

3. TeraSky: TeraSky is a prominent global provider of advanced technology solutions that incorporates Antrea into both internal Kubernetes clusters and those of various clients. The adoption of Antrea is instrumental in fortifying TeraSky's Kubernetes environment with robust and adaptable security models. The company places significant emphasis on leveraging Antrea Cluster Network Policies, Antrea Network Policies, and the Egress functionality to ensure a resilient and flexible operational framework.

### Related Projects and Vendors:

1. [Calico](https://docs.tigera.io/archive/v3.1/introduction/): Calico is a widely adopted open-source networking and network security solution for containers. It uses a combination of BGP (Border Gateway Protocol) and iptables for its data plane. Calico operates at the network layer and does not require an overlay network.
Difference between Antrea and Calico:
* Antrea: Antrea provides comprehensive security policies, including Antrea Cluster Network Policies and Antrea Network Policies. These policies enable users to define and enforce security rules at the pod level, offering granular control over network traffic.
* Calico: Calico implements Kubernetes Network Policies, which allow users to define and enforce rules for pod-to-pod communication. Calico policies are based on labels and are applied at the network layer.

2. [Cilium](https://cilium.io/):Cilium is an open source, cloud native solution for providing, securing, and observing network connectivity between workloads, fueled by the revolutionary Kernel technology eBPF
Difference between Antrea and Cillium:
* Antrea: Antrea provides Kubernetes-native security policies, including Antrea Cluster Network Policies and Antrea Network Policies. These policies allow users to define and enforce security rules at the pod level.
* Cilium: Cilium offers a range of security features, including Layer 7 (application layer) visibility and security policies. It can enforce security policies based on application awareness and API-aware network security.

3. [Flannel](https://github.com/flannel-io/flannel): Flannel is a simple and lightweight overlay network that provides basic networking connectivity between pods in a Kubernetes cluster.
Difference between Antrea and Flannel:
* Antrea: Antrea places a strong emphasis on security. It provides Kubernetes-native security policies, including Antrea Cluster Network Policies and Antrea Network Policies, allowing for fine-grained control over pod-to-pod communication.
* Flannel primarily focuses on basic networking and lacks advanced security features. It provides basic isolation between pods but doesn't have built-in support for advanced security policies.

4. [Weave](https://www.weave.works/docs/net/latest/overview/): Weave is an overlay network that operates at Layer 2 and creates a virtual network connecting containers across different hosts. It uses VXLAN encapsulation by default.
Difference between Antrea and Weave:
* Antrea: Antrea places a strong emphasis on security. It provides Kubernetes-native security policies, including Antrea Cluster Network Policies and Antrea Network Policies, allowing for fine-grained control over pod-to-pod communication.
* Weave: Weave focuses on basic networking capabilities and does not offer Kubernetes-native security policies out of the box. It provides basic network isolation between containers.

5. [Kube-router](https://www.kube-router.io/): Kube-router is a lightweight and highly configurable CNI (Container Network Interface) plugin for Kubernetes. It operates at the network layer and is designed to be modular and extensible.
Difference between Antrea and Kube-router:
* Antrea: Antrea places a strong emphasis on security. It provides Kubernetes-native security policies, including Antrea Cluster Network Policies and Antrea Network Policies, allowing for fine-grained control over pod-to-pod communication.
* kube-router: kube-router focuses on simplicity and efficiency in networking. It may not provide the same level of advanced security features as Antrea but offers basic networking isolation.

6. [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni): Multus-CNI is a multi-network plugin for Kubernetes that enables the attachment of multiple network interfaces to pods. It is designed to address scenarios where a pod needs to connect to multiple networks concurrently, each serving a specific purpose.
Difference between Antrea and Multus-CLI:
* Antrea: Antrea uses Open vSwitch (OVS) for its data plane and provides advanced networking capabilities, including support for various encapsulation modes (e.g., VXLAN, GRE, STT) and security features.
* Multus-CNI: Multus-CNI is focused on enabling the attachment of multiple network interfaces to a single pod. It doesn't provide the same level of advanced networking features as Antrea but addresses the use case of pods needing connectivity to multiple networks.

| Project | Key datapath technologies | Windows support | K8s NP | Advanced NP | BGP support | kube-proxy replacement | CNCF status | Comments |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Antrea | OVS | Yes | Yes | Yes (ACNP, ANP, Tiers) | No | Yes | Sandbox | Focus on security and some advanced networking features such as Egress (SNAT policies for Pods) and multicast. |
| Calico | iptables / eBPF | Yes | Yes | Yes. Also support for Host NetworkPolicies (not supported yet by Antrea, on roadmap) | Yes | Yes | N/A | Well-established network plugin (one of the earliest one). Several security features are only available in commercial version. |
| Cilium | eBPF | No | Yes | Yes. Also support for Host NetworkPolicies. | Yes | Yes | Graduated | Lots of interest in Cilium at the moment. Very rich ecosystem of tools, such as Hubble, Tetragon, to provide visibility into the network & applications |
| Flannel | | Yes (maintained?) | No | No | No | No | N/A | Emphasis on simplicity, not many advanced features. L3 networking. |
| Weave | iptables | No | Yes | No | No | No | N/A | L2 networking with overlay and multicast support. **No longer actively maintained** |
| kube-ovn | OVS / OVN | Yes | Yes | No | Yes | Yes | Sandbox | Focus on advanced networking features and multi-tenant networking. |
| kube-router | iptables / IPVS | No | Yes | No | Yes | Yes | N/A | Aims at networking simplicity and a modular infrastructure (pick which networking functions you need using flags). No advanced security. |
| ovn-kubernetes | OVS / OVN | Yes | Yes | No | No (unsure) | No | N/A | Mostly relevant in the context of the OpenShift K8s distribution. Advanced networking thanks to OVN (multicast, Egress). |

All projects use basic Linux networking functions such as routing as part of their datapath, so I am not pointing this out in the table (as a key datapath technology).