Antrea-LIVE notes!

# Antrea-LIVE notes! # Episode 4 ``` "group": "discovery.k8s.io", "version": "v1beta1", "kind": "EndpointSlice", "description": "EndpointSlice represents a subset of the endpoints that implement a service.\nFor a given service there may be multiple EndpointSlice objects, selected by\nlabels, which must be joined to produce the full set of endpoints.", "introduced_version": { "version_major": 1, "version_minor": 16 }, "deprecated_version": { "version_major": 1, "version_minor": 21 }, "removed_version": { "version_major": 1, "version_minor": 25 }, "replacement": { "group": "discovery.k8s.io", "version": "v1", "kind": "EndpointSlice" } }, ``` ## problem ``` // // +k8s:prerelease-lifecycle-gen:introduced=1.19 // // +k8s:prerelease-lifecycle-gen:deprecated=1.22 // // +k8s:prerelease-lifecycle-gen:removed=1.25 ``` is not in the swager spec!!!!! ## solution kube-pug parses these, and uses them (instead of swagger comments) to build a data model of semver -> API version ## Details https://deprecations.k8s.churc <-- new Kube Pug output ``` GOPATH=$(pwd) go get k8s.io/api GOPATH=$(pwd) ../generator-bin -i k8s.io/api/./... -v 9 > ../generator/html/src/data.json ``` ... why is this beter than swaggerspec? ... - smaller only deprecated APIs - NOT THE SAME AS `//go:generate ....` - https://github.com/rikatz/kubepug/tree/main/generator - - staging.src.k8s.io code-generator --> borrowed into kubepug ``` // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object // +k8s:prerelease-lifecycle-gen:introduced=1.2 // +k8s:prerelease-lifecycle-gen:deprecated=1.8 // +k8s:prerelease-lifecycle-gen:removed=1.16 // +k8s:prerelease-lifecycle-gen:replacement=apps,v1,ReplicaSetList ``` filed ! https://github.com/kubernetes-sigs/external-dns ``` TargetSelector string `json:"targetSelector,omitempty" protobuf:"bytes,3,opt,name=targetSelector"` } // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object ``` // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object ``` apiVersion: externaldns.k8s.io/v1alpha1 <--- should be fixed??? kind: DNSEndpoint metadata: name: examplemxrecord spec: endpoints: - dnsName: example.com recordTTL: 180 recordType: MX targets: - 10 mailhost1.example.com - 20 mailhost2.example.com ``` - kubepug - swagger ## example https://raw.githubusercontent.com/rikatz/kubepug/main/test/testdata/manifests/psp1.yaml ## https://deprecations.k8s.church/ ~~~ How new K8s deprecations work ~~~ ## problem - non standard deprecations metadata in k8s Swagger API spec ## Deprecated api - ➜ 41 wget https://raw.githubusercontent.com/kubernetes/kubernetes/v1.21.0/api/openapi-spec/swagger.json - downloads https://github.com/rikatz/kubepug/blob/main/pkg/utils/downloader.go - parses the API ~ deprecations ``` 5594 }, 5595 "io.k8s.api.core.v1.ComponentStatus": { 5596 "description": "ComponentStatus (and ComponentStatusList) holds the cluster validation info. Deprecated: This API is deprecated in v1.19+", 5597 "properties": { 5598 "apiVersion": { 5599 "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architectur e/api-conventions.md#resources", 5600 "type": "string" 5601 }, 5602 "conditions": { 5603 "description": "List of component conditions observed", ``` # Episode 37 Luther spectrocloud is here CAPI https://github.com/kubernetes/kubernetes/issues/22469 https://github.com/kubernetes/kubernetes/pull/24154/files v1beta api const ( baseURL = "https://raw.githubusercontent.com/kubernetes/kubernetes" fileURL = "api/openapi-spec/swagger.json" ) ## April 2016 872d2b71c63431878e382e6e34c581c047f04791 "" - docs/proposals/network-policy.md - NetworkPolicySpec - HAD - PodSelector - Ingress - MISSING: - Egress, - PolicyTypes - Question: If we knew there would be i/e ... would we have made it a "policyTypes", or rather, just made "type" - Match NOTHING `ingress: []`, match EVERYTHING `ingress: - {}`, default is "Ingress" for historical reasons.. - If they started w/ ing/egress, probably, it'd be required?! - Possibility suggested as podSelectors to be NESTED in ingress rules - Thread https://groups.google.com/g/kubernetes-sig-network/c/vXWUNpkiJ30/m/WoGHQb01AwAJ ``` IIRC we stuck with a whitelist because we wanted to get a baseline policy into Kubernetes earlier, and argue about more complex policy like this later :) ``` And ``` wanted to keep a flatter mapping between NP objects and pods, instead of huge NP objects that have a ton of rules that can apply to many different things ``` CODE: staging/src/k8s.io/api/networking/v1/types.go ## First bug fix , 2016 6fe7da72b4b19abb900cf0e3ba0c84314abb7d7f /pkg/apis/extensions/validation/validation.go is where K8s APIs get validated Current code --> ``` allErrs = append(allErrs, apivalidation.ValidateObjectMetaUpdate(&update.ObjectMeta, &old.ObjectMeta, field.NewPath("metadata"))...) allErrs = append(allErrs, ValidateNetworkPolicySpec(&update.Spec, field.NewPath("spec"))...) ``` Lets check an error `CLUSTER=antrea ./kind-local-up.sh` ... https://github.com/jayunit100/k8sprototypes Let's trigger a validation.go error... ``` kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: web-deny-all spec: podSelector: matchLabels: app: web ingress: - from: ports: - protocol: TCP port: 22222 endPort: 11111 ``` ./pkg/apis/networking/validation/validation.go error !!! ## Future... ### 2017 -> 2020 - calico and cillium implements policy fully (Summer 2017) - - https://github.com/cilium/cilium/releases?page=20 - https://github.com/projectcalico/calico/releases?page=19 - https://github.com/projectcalico/canal/blob/master/Canal%20Phase%201%20Diagram.png canal , allowing flannel and calico to work together for policies - September 29, 2017 GKE Announce calico support - Emerging pattern: Cloud native CNI w/ a OSS networkpolicy provider on top. - Feb 1, 2020 - Creation of networkpolicy working group https://groups.google.com/g/kubernetes-sig-network/c/bj3FNFKeHew/m/kvxxSHxMAQAJ - Antrea introduced (Aug 2020) - https://blogs.vmware.com/networkvirtualization/2020/08/container-networking-antrea.html/ - https://techcrunch.com/2020/11/10/with-29m-in-funding-isovalent-launches-its-cloud-native-networking-and-security-platform-based-on-ebpf-and-cilium/ - found Interesting caveats, like loopback policies (oct 26, 2020) (https://github.com/cilium/cilium/issues/13767) ### 2021 NetworkPolicy Conformance and Community building https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1611-network-policy-validation https://kubernetes.io/blog/2021/04/20/defining-networkpolicy-conformance-cni-providers/ Running the netpol e2e's ``` /home/ubuntu/SOURCE/kubernetes/e2e.test --provider=local --kubeconfig=/home/ubuntu/.kube/config --dump-logs-on-failure=false --ginkgo.focus="NetworkPolicy" --ginkgo.skip="Driver|Slow|Driver" ``` https://github.com/kubernetes/kubernetes/tree/master/test/e2e/network/netpol ### After 2021+ , reaping rewards - https://docs.google.com/document/d/1_clStao-uM3OblOTsA4Kgx2y4C9a6KXmdOywW4tFSLY/edit#heading=h.bo78zxbiay3b - https://github.com/kubernetes/enhancements/tree/master/keps/sig-network - validation - port-range - admin-network-policy - ntwork-policy status - https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/2091-admin-network-policy <-- # Episode 36 rustletttt News: - Netpol WG update - implicit deny -- no whitelist model - serviceaccount selector bc adding new fields not backward compatible - silent errors result security holes ``` // // Exactly one field must be set. <--- // // +kubebuilder:validation:MaxProperties=1 type AdminNetworkPolicySubject struct { // if a CNI doesnt understand this field - it could fail closed ``` kube builder validation fixes it https://docs.krustlet.dev/ https://www.solo.io/products/web-assembly/ - write in rust | write in any lang - put binary in registry | put container in registry - run in rustlet | run on kubelet ``` curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` ### Multicluster policies - cillium clustermesh - submariner (l3) + scupper (l7) - antrea multi-cluster policy ### rust term - cargo <-- go mod - crates <-- "module package" - unmaintained as of now... june 25th - needs an azure registry to run (cant use docker.io) # Episode 32 - https://github.com/astoycos/go-ebpf-proxy-example - https://github.com/kubernetes-sigs/kpng/blob/master/global-state.yaml - https://github.com/kubernetes-sigs/kpng/blob/master/doc/arch.svg - https://github.com/kubernetes-sigs/kpng/pull/217/files - https://yawn.io/jekyll/update/2020/02/19/turing-complete.html # Episode 29 - protobuf: serialization/wire format --> IDL - grpc: http framework that does RPCs using protobuf - Kubernetes - CSI: native to the specification - CNI: ? not yet 2.0 ??? exec - Kubelet: - services, messages, enum - service - endpoint - watch - ... ``` rpc Watch() returns (stream OpItem); ``` - containerd [exec] -> /opt/cni/bin/antrea -> stdout , IP parsed ? - /opt/cni/bin/antrea - ( GRPC ) -> antrea-agent (on a port) ``` pkg/kubelet/cm/devicemanager/manager.go ``` - vlad - csv line --> new protobuf - https://github.com/grpc/grpc-go/blob/master/dialoptions.go # Episode 27???? - chris (grace) going to show us AKO stuff moving forward ! ## 1.24 - kubelet_volume_stats_health_status_abnormal, pvc1 - kubelet_volume_stats_health_status_abnormal, pvc2 - kubelet_volume_stats_health_status_abnormal, pvc3 <-- - ... - whats this `CSIMigration` feature gate all about? - t shirt ... ``` Up until now, Kubernetes automatically created a service account Secret when creating a Pod. That token Secret contained the credentials for accessing the API. ``` - NetworkPolicyStatus https://github.com/antrea-io/antrea/pull/1442 - https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/1959-service-lb-class-field/README.md ## Writing the KPNG kep - This is the live notes page for antrea..... # Antrea LIVE Episode 26 etcd and CRDs and stuff ## THE FIRST ANT LIVE CODING CHALLENGE EVER MAKE THIS REPRODOCE https://github.com/kubernetes/kubernetes/issues/65517 ``` #!/bin/bash #NAME=yyy kubectl apply -f <(cat << EOF apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: foo${NAME}s.stable.example.com spec: group: stable.example.com versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: scope: Namespaced names: plural: foo${NAME}s singular: foo${NAME} # kind is normally the CamelCased singular type. Your resource manifests use this. kind: Foo${NAME} shortNames: - foo${NAME}s EOF ) kubectl get foo${NAME}s echo 2 kubectl get foo${NAME}s echo 3 kubectl get foo${NAME}s kubectl delete CustomResourceDefinition foo${NAME}s.stable.example.com ``` # Antrea LIVE Episode 24 etcd , ricardo, ... https://github.com/ahrtr/etcd-issues/tree/master/issues/13766 ``` while true ; do etcdctl --endpoints="https://localhost:2379" --cacert="/etc/kubernetes/pki/etcd/ca.crt" --cert="/etc/kubernetes/pki/etcd/server.crt" --key=/etc/kubernetes/pki/etcd/server.key endpoint status --cluster ; sleep 1 ; done ``` perf ``` etcdctl --endpoints="https://localhost:2379" --cacert="/etc/kubernetes/pki/etcd/ca.crt" --cert="/etc/kubernetes/pki/etcd/server.crt" --key=/etc/kubernetes/pki/etcd/server.key check perf --load=xl ``` # Antrea LIVE Episode 21 Introduction of antrea policy: - what it is - how to use it - how to debug it # Antrea LIVE Episode 20 ## News Antrea 1.5.1 - https://github.com/antrea-io/antrea/releases/tag/v1.5.1 - Azure - TKG CAPZ w/ Antrea default installation - [x] https://github.com/vmware-tanzu/tanzu-framework/blob/main/pkg/v1/providers/infrastructure-azure/v1.0.1/ytt/base-template.yaml#L22 - [x] Configmap `AntreaNetworkPolicy: false` - [x] Tests a few antrea netpol - Antrea AKS installation - [ ] https://antrea.io/docs/v1.5.1/docs/aks-installation/ - [ ] FULL replacement of Azure CNI -> Antrea (Netpol + CNI/IPAM) - AKS-Engine (deprecated) can provide Netpol on Antrea and IPAM + routing on Antrea CNI plugins. - [ ] Explore both /etc/cni/ | /opt/cni - [ ] trafficMode=networkpolicyonly - [ ] Looking code on Azure CNI # Antrea LIVE Episode 19 Introduction of antrea egress and demo how to use it # Antrea LIVE Episode 18 - Elenaor ! wants to play w/ the application path - velero, sonobuoy, ... - GKE external loadbalancing ? - kubernetes pods w/ VRFs - cant use AVI - cant use k8s services - ... - VRFs vs overlays - vrf: hardware routers etc - vrf: not for modern apps # Antrea LIVE Episode 17 - Lets break the CNI down W/ Mike Zappa - Quick CNI Review (Inputs, Outputs, ENV VARS) - Diagram of what we are going to do build - Workflow End to End Kubectl -> Bridge/Host-Local CNI Plugin - ADD/DEL/CHECK - Demo's - K8s manual setup - basic shell cni plugin - cnitool - BGP # Antrea LIVE Episode 16 ### News [Containerd 1.6 - HostProcess Container support for Windows](https://github.com/containerd/containerd/releases/tag/v1.6.0) - [Kubernetes Policy Management Whitepaper](https://github.com/kubernetes/sig-security/blob/main/sig-security-docs/papers/policy/CNCF_Kubernetes_Policy_Management_WhitePaper_v1.pdf) - [NetworkPolicy Status KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/2943-networkpolicy-status) ``` Conditions []metav1.Conditon ``` - antrea cni 0.3.0 -> 0.4.0 (CHECK) ? - `kubectl delete pod -n kube-system -l "component=antrea-agent"` - containerd -> go-cni - runc - containerd (go-cni) cni add - co-cni imports. github.com/containernetworking/cni v1.0.1 - parses the cni json config - matches 0.4.0 to the versions of its own cni config parser ## Links https://www.cni.dev/docs/spec/ # Antrea LIVE Episode 15 ### Routable Pods - Scott Rosenberg shares about nsx integration with antrea on tkgm for routable pods ### Kubebuilder and lb operator for k8s - Xinqi gives a brief introduction about kubebuilder and then take a look at a real operator "load balancr operator for kubernetes" # Antrea LIVE Episode 14 ### K8s news - Aiden Obeley is from philly - Interesting IPVS question https://kubernetes.slack.com/archives/C09QYUH5W/p1643248958272400 ``` I have some questions about why kube-proxy with --proxy-mode=ipvs would always /d a LoadBalancer Service's ExternalIP to the dummy IPVS interface (kube-ipvs0). We noticed this while changing kube-proxy (v1.20.11) from --proxy-mode=iptables to --proxy-mode=ipvs. When the IPVS Proxier handle LB Service, it calls syncService() and always bind the LB's ExternalIP to the kube-ipvs0 dummy interface. Meanwhile, there doesn't seem to be similar things done in iptables Proxier. I'd like to understand why the binding is assumed by default. Is it meant to hairpin/SNAT traffic from the nodes that would go to the LoadBalancer? ``` - https://github.com/kubernetes/enhancements/tree/master/keps/sig-multicluster/1645-multi-cluster-services-api?utm_source=thenewstack&utm_medium=website&utm_campaign=platform # Antrea LIVE Episode 13 ### Antrea News - Antrea v1.5 is out! [release notes](https://github.com/antrea-io/antrea/releases/tag/v1.5.0) - Highlights of this release include the addition of multi-cluster functionalities, multicast support and improvements to Antrea IPAM ### AVI Software Load Balancer and Ingress Gateway - What is expected out of enterprise grade k8s ingress - Avi LB and Ingress solution - Technical deep dive and demo - Global load balancing for multi cluster(DNS based) ### AVI in Tanzu - Create a vSphere cluster with ako installed using Tanzu - Deep dive into the structure of ako-operator - Use AVI as control plane High Availability provider - Separate the vip network of load balancer type of service ### Audience Q's - audience question calico vs antrea # Antrea LIVE Episode 12 ### NOTES: - antrea 1.5 release !!! -> - [why-we-need-lens-as-a-kubernetes-ide](https://containerjournal.com/editorial-calendar/best-of-2021/why-we-need-lens-as-a-kubernetes-ide - [5 Kubernetes trends to watch in 2022](https://enterprisersproject.com/article/2022/1/5-kubernetes-trends-watch-2022) - https://github.com/bentoml/BentoML - QUESTION: How does NSX / antrea hook interoperate over OVS ? ### k8s-service-validator - problem statement - purpose and soluton - CNIs - pod-to-pod connectivity matrix - run - integration with sonobuoy - run on your laptop - code walkthrough - how to build your test with our validator? an example: ProxyNameLabel - unit tests - welcome contributers - Github: https://github.com/K8sbykeshed/k8s-service-validator # Antrea LIVE: Episode 11 - NEWS - [Reserve Service IP Ranges For Dynamic and Static IP Allocation](https://github.com/kubernetes/enhancements/issues/3070) - [How to Use the Windows CSI Proxy and CSI SMB Driver for Kubernetes](https://www.phillipsj.net/posts/how-to-use-the-windows-csi-proxy-and-csi-smb-driver-for-kubernetes/) - [Kubernetes Release 1.24 schedule](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.24) - [Kubernetes is Moving on From Dockershim](https://kubernetes.io/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/) - [SIG Windows DEV tools](https://github.com/kubernetes-sigs/sig-windows-dev-tools) (jay) * [x] How it's installed * Makefile targets * [x] Machines bootstrap * Vagrant provision * image-builder vbox hypervisor [[1]](#link-1) * support for images used downstream by ClusterAPI - [Deep Dive Antrea CNI on Windows](https://github.com/antrea-io/antrea/blob/main/docs/windows.md) (amim) * [ ] setup files: forked/[0-1]-antrea.ps1 * challenges in the node * [x] OVS Windows binaries debugging (thanks [Alin](https://twitter.com/aserdean_)) * Missing DLLs [[2]](#link-2) * `Invoke-Item ovsdb-server.exe` * [x] Multiple NICs, Kubelet and CNI overlay configs * [x] Windows: when `kubectl describe` isnt enough... * Not so descriptive failure: `Unknown desc = failed to reserve sandbox name` * Logs from `containerd` goes to ETW (event tracing for windows) - how to extract them? * ETW output how to extract? [[3]](#link-3) * `sc.exe config containerd binPath="c:\Program Files\containerd\containerd.exe --run-service --log-file=c:\tmp\containerd.log"` * Ops - `failed to find plugin \"portmap\" in path [C:/opt/cni/bin]` * Wrong (Linux) CNI configuration downloaded :) * PS C:\etc\cni\net.d> cat .\10-antrea.conflist - Kubernetes Services Session Affinity * [ ] Services and Endpoints * [ ] ClusterIP vs. NodePort * Expectation of load balancing on both services types. * [ ] Antrea-Proxy and services types * ClusterIP -> Antrea-proxy * NodePort -> Kube-proxy userspace [[4]](#link-4) * [ ] SessionAffinity and TTL configuration * Sticky client connection to the same pod. * `clientIP.timeoutSeconds` - set the maximum session sticky time. * Two odd issues on winuserspace: 1. New requests zero the expiration counter. In constant traffic flow, it never expires. 2. ttlSeconds is ignored on endpoint update. [[5]](#link-5) * [ ] Antrea NodePortLocal [[6]](#link-6) - Next steps * [ ] Kube-proxy [win]userspace deprecation [[7]](#link-7) * [ ] Antrea-proxy and Antrea Agent features!! * [ ] Windows Operational Readiness KEP [[8]](#link-8) * [ ] [kPNG](https://github.com/kubernetes-sigs/kpng/) ## Links [1] https://github.com/kubernetes-sigs/sig-windows-dev-tools/tree/master/experiments/image-builder [2] https://www.dependencywalker.com/ [3] https://github.com/kubernetes-sigs/sig-windows-tools/pull/197 [4] https://thefind.live/posts/winuserspace/readme/ [5] https://github.com/kubernetes/kubernetes/pull/107399/ [6] https://antrea.io/docs/main/docs/node-port-local/ [7] https://github.com/kubernetes/kubernetes/issues/103860 [8] https://github.com/kubernetes/enhancements/pull/2975 https://github.com/nicolaka/netshoot ### Commands #### Disabled remote desktop firewall `Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server'-name "fDenyTSConnections" -Value 0` `Enable-NetFirewallRule -DisplayGroup "Remote Desktop"` # Antrea LIVE: Episode 10 ## NEWS - how tanzu featuregates work https://github.com/vmware-tanzu/tanzu-framework/pull/1370 - Hacking tanzu by pausing reconcilation controllers ## Links - [NSX-T 3.2 integration with Antrea](https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/administration/GUID-9197EF8A-7998-4D1B-B968-067007C56B5C.html) - [New imgpkg version](https://github.com/vmware-tanzu/carvel-imgpkg/releases/tag/v0.24.0) - [Helm To Carvel Conversion Tool](https://github.com/TeraSky-OSS/helm-to-carvel-conversion-tool) - [Terraforming Tanzu](https://github.com/tanzuformers) - [Updated TKGm Customization Repo](https://github.com/vrabbi/tkgm-customizations) * [Added Cilium CNI](https://github.com/vrabbi/tkgm-customizations/tree/master/tkg-customizations/custom-ytt-overlays/cluster-config-changes/cilium) * [Added Antrea + NSX-T Integration](https://github.com/vrabbi/tkgm-customizations/tree/master/tkg-customizations/custom-ytt-overlays/cluster-config-changes/antrea-nsx-integration) * [Added Calico with Typha and BGP Configuration](https://github.com/vrabbi/tkgm-customizations/tree/master/tkg-customizations/custom-ytt-overlays/cluster-config-changes/custom-calico) * [Added KubeProxy IPVS config](https://github.com/vrabbi/tkgm-customizations/tree/master/tkg-customizations/custom-ytt-overlays/cluster-config-changes/ipvs) * [Added overlay to include Thanos in TKG Prometheus package](https://github.com/vrabbi/tkgm-customizations/tree/master/tkg-customizations/tkg-extension-modifications/packages/prometheus/thanos-sidecar) ## Show Notes # Antrea LIVE: Episode 9 ## NEWS Whats it like to maintain an ingress/gateway provider? Featuring Ricardo ! - history lesson - the eye of horus is on the dollar bill - ingress nginx: - first controller, ricardo steps in for alejandro - AVI - LIMITED gateway support in avi right now - experimental for customers - GSLB : *GLobal Server Loadbalancing* - GSLB and Avi Ingress Lab https://labs.hol.vmware.com/HOL/catalogs/lab/10883 - Contour - - NGINX - Envoy - ... # Antrea LIVE: Episode 8 - multiple clusters, support global services visited by different member clusters. - multicast - flexible ipam - per namespace - per pod <-- new + persistant IPs - questions from KubeCon China: @vicky - antrea vs kube-ovn ~> - antrea vs cillium ~> - performance improvements ~> - new features ~> # Antrea LIVE: Episode 7 (CAPA cniIngressRules, ) SPECIAL GUEST @sedef savas !!! AND DWAYNE!!!! - Sedef is o.g. from the original TKG team - https://kubernetes.io/blog/2021/12/08/dual-stack-networking-ga/ - https://github.com/kubernetes-sigs/kpng/issues/142 Readyness states - https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/discovery/types.go - CAPA - declarative cluster / infra mgmt - https://github.com/kubernetes-sigs/cluster-api-provider-aws - https://cluster-api-aws.sigs.k8s.io/ - same mgmt cluster - different credentials - what are secondary cidrblocks for??? - classic Loadbalancers - transient gateways - hub and spoke peering - AWS Networking Details - NAT gateways - private Epis - public subnets - Networking.V2 - VPC-peering - Security Groups - IAM resources and CAPA # Antrea LIVE: Episode 6 (Antrea Agent on K3s, Windows, GKE, ... ) - luther is here again ! - k3s demo - test networkpolicys - https://github.com/kubernetes/enhancements/pull/2975 - https://github.com/rancher/rke2/issues/2201 - https://github.com/kubernetes-sigs/sig-windows-dev-tools/blob/master/forked/0-antrea.ps1 # Antrea LIVE: Episode 5 (Carvel and the CNI Lifecycle inside Tanzu Framework) - api deprecations - https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/ - https://github.com/vmware-tanzu/tanzu-framework/pull/959/ - https://github.com/vmware-tanzu/tanzu-framework/pull/959/, addon reconcilers - How tanzu addons for CNI, etc, work - kubetail on https://github.com/vmware-tanzu/tanzu-framework/tree/main/addons#workflow-of-tanzu-addons-manager - https://carvel.dev/kapp-controller/docs/latest/packaging/ - PackageRepository - PackageMetadata - Package - PackageInstall - note the `cluster` field... its multicluster by default - kubectl edit pkg cert-manager.tanzu.vmware.com.1.5.3+vmware.2-tkg.1-zshippable - goes to custom APIServer extension - not a CRD - Managed by kapp controller # Antrea LIVE: Episode 4 (AntreaProxy, 1.4, and ) - serviceAccount selectors for networkPolicies ! - egress IPs https://antrea.io/docs/v1.3.0/docs/egress/ <-- in next version of vmware tanzu (1.2.3, ipv4) - Installing Antrea 1.4 - AntreaProxy Enablement - https://github.com/antrea-io/antrea/pull/2632 <-- doc updates for antreaproxy - Running Antrea w/o kube proxy - does it work ? - VMWare Tanzu feature gates for CNI providers and the Carvel tooling - kapp - ytt - kapp-controller - BONUS ! looking at @aojeas Conntrack fix - https://github.com/kubernetes/kubernetes/issues/105657 - https://github.com/kubernetes/kubernetes/pull/106163/files ``` root@antrea-control-plane:/# ovs-ofctl dump-flows br-int table=41 cookie=0x1040000000000, duration=2389.591s, table=41, n_packets=0, n_bytes=0, idle_age=2389, priority=200,tcp,reg4=0x10000/0x70000, nw_dst=10.96.85.206, <-- CLUSTER IP OF NGINX tp_dst=80 actions=load:0x2->NXM_NX_REG4[16..18],load:0x1->NXM_NX_REG0[19],load:0x6->NXM_NX_REG7[], group:6 <-- a definition in OVS ``` ``` root@antrea-control-plane:/# ovs-ofctl dump-groups br-int 6 NXST_GROUP_DESC reply (xid=0x2): group_id=6, type=select, bucket=bucket_id:0, <-- buckets? weight:100, actions=load:0xa0a0204->NXM_NX_REG3[], load:0x50->NXM_NX_REG4[0..15], resubmit(,EndpointDNAT) ``` # Antrea LIVE: Episode 3 (Multus, Whereabouts, and host-local IPAM) - host ports on windows, do they work? or do you need nodeportlocal - IF -> elastic IP (aws CNI) -> limited - encap - `service-proxy-name` <-- annotation on k8s services that disables proxying - why multiple nics - bypass CNI network for perf, minimize hops - sr-iov/dpdk/edp - multus quickstart: - hostlocal allocation per node - skip dchp, allocate from a pool - fork of `static` plugin - shim + thick component (k8s controller) - sriov has a device plugin. # Show Details Show link: https://youtu.be/Q1CBFoMAG2g Live notes: https://hackmd.io/wxNOmhZdRNm_hJzZNFKwRg https://github.com/antrea-io/antrea/tree/main/docs/cookbooks/multus ## notes from jiunjen: - CNF/secondary network support in Antrea in collaboration with Intel ... SR-IOV is here: https://github.com/antrea-io/antrea/pull/2651 - Native support means it (no external CNIs required) # Episode 2 ## Show Details # Antrea LIVE: Episode 2 (antrea CNI and ovs metrics + a new sessionAffinity kubeproxy bug) # Show Details Show link: https://www.youtube.com/watch?v=3aUnws6diAY ## Antrea stuff - vivek, multus, and multus+telco networking - https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md - Connection tracking (“conntrack”) is a core feature of the Linux kernel's networking stack. It allows the kernel to keep track of all logical network connections or flows, and thereby identify all of the packets which make up each flow so they can be handled consistently together - conntrack is leveraged by stateful services, like stateful firewall, NAT, that require to track connections. The connections are not just TCP, and can be UDP, ICMP, SCTP, etc. - Flow record, 2 records per connection, aggregate - ELK Stack super cool; Antrea layer 4 service mesh - future: policy recommendation engines + performance analytics - prometheus installation and OVS monitoring ``` $> kubectl apply -f build/yamls/antrea-prometheus.yml $> sleep 60 ; kubectl port-forward pod/prometheus-deployment-79fb7d997f-nkbjj -n monitoring --address 0.0.0.0 9090:9090 -n monitoring ``` - table 90 - https://github.com/antrea-io/antrea/blob/main/docs/design/ovs-pipeline.md - https://github.com/antrea-io/antrea/blob/main/docs/prometheus-integration.md ## New in upstream - a new kube-proxy affinity bug https://github.com/kubernetes/kubernetes/pull/106030 - https://github.com/kubernetes/kubernetes/issues/105657 - https://github.com/K8sbykeshed/k8s-service-lb-validator/issues/35 ---- # Episode 1 ## Show Details https://www.youtube.com/watch?v=aWUwxQ58bEQ ### Antrea 1.3.0 - kubectl apply -f https://github.com/antrea-io/antrea/releases/download/v1.3.0/antrea.yml - install ~ jayunit100/k8sprototypes/kind/ kind-local-up.sh ### FQDN Policies ``` apiVersion: crd.antrea.io/v1alpha1 kind: ClusterNetworkPolicy metadata: name: acnp-fqdn-all-foobar spec: priority: 1 appliedTo: - podSelector: matchLabels: app: client egress: - action: Drop to: - fqdn: "*foobar.com" ``` ## K8sNetLook - https://github.com/sarun87/k8snetlook ## Notes - Release Notes: https://github.com/antrea-io/antrea/releases/tag/v1.3.0 - https://github.com/kubernetes/enhancements/pull/2522 - `kubectl get acnp` - `CLUSTER=antrea CONFIG=kind-conf.yaml ./kind-local-up.sh` - https://kubernetes.io/docs/concepts/services-networking/network-policies/ - https://github.com/sarun87/k8snetlook/blob/master/examples/run-k8s.yaml - bitnami/nginx image to avoid docker limits - Arun's awesome kubecon talk slides: https://kccncna2021.sched.com/event/lV0b

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.