owned this note
owned this note
Published
Linked with GitHub
# Antrea-LIVE notes!
# Episode 4
```
"group": "discovery.k8s.io",
"version": "v1beta1",
"kind": "EndpointSlice",
"description": "EndpointSlice represents a subset of the endpoints that implement a service.\nFor a given service there may be multiple EndpointSlice objects, selected by\nlabels, which must be joined to produce the full set of endpoints.",
"introduced_version": {
"version_major": 1,
"version_minor": 16
},
"deprecated_version": {
"version_major": 1,
"version_minor": 21
},
"removed_version": {
"version_major": 1,
"version_minor": 25
},
"replacement": {
"group": "discovery.k8s.io",
"version": "v1",
"kind": "EndpointSlice"
}
},
```
## problem
```
// // +k8s:prerelease-lifecycle-gen:introduced=1.19
// // +k8s:prerelease-lifecycle-gen:deprecated=1.22
// // +k8s:prerelease-lifecycle-gen:removed=1.25
```
is not in the swager spec!!!!!
## solution
kube-pug parses these, and uses them (instead of swagger comments)
to build a data model of semver -> API version
## Details
https://deprecations.k8s.churc <-- new Kube Pug output
```
GOPATH=$(pwd) go get k8s.io/api
GOPATH=$(pwd) ../generator-bin -i k8s.io/api/./... -v 9 >
../generator/html/src/data.json
```
... why is this beter than swaggerspec? ...
- smaller only deprecated APIs
-
NOT THE SAME AS `//go:generate ....`
- https://github.com/rikatz/kubepug/tree/main/generator
-
- staging.src.k8s.io code-generator --> borrowed into kubepug
```
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// +k8s:prerelease-lifecycle-gen:introduced=1.2
// +k8s:prerelease-lifecycle-gen:deprecated=1.8
// +k8s:prerelease-lifecycle-gen:removed=1.16
// +k8s:prerelease-lifecycle-gen:replacement=apps,v1,ReplicaSetList
```
filed ! https://github.com/kubernetes-sigs/external-dns
```
TargetSelector string `json:"targetSelector,omitempty" protobuf:"bytes,3,opt,name=targetSelector"`
}
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
```
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
```
apiVersion: externaldns.k8s.io/v1alpha1 <--- should be fixed???
kind: DNSEndpoint
metadata:
name: examplemxrecord
spec:
endpoints:
- dnsName: example.com
recordTTL: 180
recordType: MX
targets:
- 10 mailhost1.example.com
- 20 mailhost2.example.com
```
- kubepug
- swagger
## example
https://raw.githubusercontent.com/rikatz/kubepug/main/test/testdata/manifests/psp1.yaml
## https://deprecations.k8s.church/
~~~ How new K8s deprecations work ~~~
## problem
- non standard deprecations metadata in k8s Swagger API spec
## Deprecated api
- ➜ 41 wget https://raw.githubusercontent.com/kubernetes/kubernetes/v1.21.0/api/openapi-spec/swagger.json
- downloads https://github.com/rikatz/kubepug/blob/main/pkg/utils/downloader.go
- parses the API ~ deprecations
```
5594 },
5595 "io.k8s.api.core.v1.ComponentStatus": {
5596 "description": "ComponentStatus (and ComponentStatusList) holds the cluster validation info. Deprecated: This API is deprecated in v1.19+",
5597 "properties": {
5598 "apiVersion": {
5599 "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architectur e/api-conventions.md#resources",
5600 "type": "string"
5601 },
5602 "conditions": {
5603 "description": "List of component conditions observed",
```
# Episode 37
Luther spectrocloud is here CAPI
https://github.com/kubernetes/kubernetes/issues/22469
https://github.com/kubernetes/kubernetes/pull/24154/files v1beta api
const (
baseURL = "https://raw.githubusercontent.com/kubernetes/kubernetes"
fileURL = "api/openapi-spec/swagger.json"
)
## April 2016
872d2b71c63431878e382e6e34c581c047f04791 ""
- docs/proposals/network-policy.md
- NetworkPolicySpec
- HAD
- PodSelector
- Ingress
- MISSING:
- Egress,
- PolicyTypes
- Question: If we knew there would be i/e ... would we have made it a "policyTypes", or rather, just made "type"
- Match NOTHING `ingress: []`, match EVERYTHING `ingress: - {}`, default is "Ingress" for historical reasons..
- If they started w/ ing/egress, probably, it'd be required?!
- Possibility suggested as podSelectors to be NESTED in ingress rules
- Thread https://groups.google.com/g/kubernetes-sig-network/c/vXWUNpkiJ30/m/WoGHQb01AwAJ
```
IIRC we stuck with a whitelist because we wanted to get a baseline
policy into Kubernetes earlier, and argue about more complex policy
like this later :)
```
And
```
wanted to keep a flatter mapping between NP objects and
pods, instead of huge NP objects that have a ton of rules that can
apply to many different things
```
CODE: staging/src/k8s.io/api/networking/v1/types.go
## First bug fix , 2016
6fe7da72b4b19abb900cf0e3ba0c84314abb7d7f
/pkg/apis/extensions/validation/validation.go is where K8s APIs get validated
Current code -->
```
allErrs = append(allErrs, apivalidation.ValidateObjectMetaUpdate(&update.ObjectMeta, &old.ObjectMeta, field.NewPath("metadata"))...)
allErrs = append(allErrs, ValidateNetworkPolicySpec(&update.Spec, field.NewPath("spec"))...)
```
Lets check an error `CLUSTER=antrea ./kind-local-up.sh` ... https://github.com/jayunit100/k8sprototypes
Let's trigger a validation.go error...
```
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: web-deny-all
spec:
podSelector:
matchLabels:
app: web
ingress:
- from:
ports:
- protocol: TCP
port: 22222
endPort: 11111
```
./pkg/apis/networking/validation/validation.go error !!!
## Future...
### 2017 -> 2020
- calico and cillium implements policy fully (Summer 2017) -
- https://github.com/cilium/cilium/releases?page=20
- https://github.com/projectcalico/calico/releases?page=19
- https://github.com/projectcalico/canal/blob/master/Canal%20Phase%201%20Diagram.png canal , allowing flannel and calico to work together for policies
- September 29, 2017 GKE Announce calico support
- Emerging pattern: Cloud native CNI w/ a OSS networkpolicy provider on top.
- Feb 1, 2020
- Creation of networkpolicy working group https://groups.google.com/g/kubernetes-sig-network/c/bj3FNFKeHew/m/kvxxSHxMAQAJ
- Antrea introduced (Aug 2020)
- https://blogs.vmware.com/networkvirtualization/2020/08/container-networking-antrea.html/
- https://techcrunch.com/2020/11/10/with-29m-in-funding-isovalent-launches-its-cloud-native-networking-and-security-platform-based-on-ebpf-and-cilium/
- found Interesting caveats, like loopback policies (oct 26, 2020) (https://github.com/cilium/cilium/issues/13767)
### 2021 NetworkPolicy Conformance and Community building
https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1611-network-policy-validation
https://kubernetes.io/blog/2021/04/20/defining-networkpolicy-conformance-cni-providers/
Running the netpol e2e's
```
/home/ubuntu/SOURCE/kubernetes/e2e.test --provider=local --kubeconfig=/home/ubuntu/.kube/config --dump-logs-on-failure=false --ginkgo.focus="NetworkPolicy" --ginkgo.skip="Driver|Slow|Driver"
```
https://github.com/kubernetes/kubernetes/tree/master/test/e2e/network/netpol
### After 2021+ , reaping rewards
- https://docs.google.com/document/d/1_clStao-uM3OblOTsA4Kgx2y4C9a6KXmdOywW4tFSLY/edit#heading=h.bo78zxbiay3b
- https://github.com/kubernetes/enhancements/tree/master/keps/sig-network
- validation
- port-range
- admin-network-policy
- ntwork-policy status
- https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/2091-admin-network-policy <--
# Episode 36
rustletttt
News:
- Netpol WG update
- implicit deny -- no whitelist model
- serviceaccount selector bc adding new fields not backward compatible
- silent errors result security holes
```
// // Exactly one field must be set. <---
// // +kubebuilder:validation:MaxProperties=1
type AdminNetworkPolicySubject struct {
// if a CNI doesnt understand this field - it could fail closed
```
kube builder validation fixes it
https://docs.krustlet.dev/
https://www.solo.io/products/web-assembly/
- write in rust | write in any lang
- put binary in registry | put container in registry
- run in rustlet | run on kubelet
```
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
### Multicluster policies
- cillium clustermesh
- submariner (l3) + scupper (l7)
- antrea multi-cluster policy
### rust term
- cargo <-- go mod
- crates <-- "module package"
- unmaintained as of now... june 25th
- needs an azure registry to run (cant use docker.io)
# Episode 32
- https://github.com/astoycos/go-ebpf-proxy-example
- https://github.com/kubernetes-sigs/kpng/blob/master/global-state.yaml
- https://github.com/kubernetes-sigs/kpng/blob/master/doc/arch.svg
- https://github.com/kubernetes-sigs/kpng/pull/217/files
- https://yawn.io/jekyll/update/2020/02/19/turing-complete.html
# Episode 29
- protobuf: serialization/wire format --> IDL
- grpc: http framework that does RPCs using protobuf
- Kubernetes
- CSI: native to the specification
- CNI: ? not yet 2.0 ??? exec
- Kubelet:
- services, messages, enum
- service
- endpoint
- watch
- ...
```
rpc Watch() returns (stream OpItem);
```
- containerd [exec] -> /opt/cni/bin/antrea -> stdout , IP parsed ?
- /opt/cni/bin/antrea - ( GRPC ) -> antrea-agent (on a port)
```
pkg/kubelet/cm/devicemanager/manager.go
```
- vlad
- csv line --> new protobuf
- https://github.com/grpc/grpc-go/blob/master/dialoptions.go
# Episode 27????
- chris (grace) going to show us AKO stuff moving forward !
## 1.24
- kubelet_volume_stats_health_status_abnormal, pvc1
- kubelet_volume_stats_health_status_abnormal, pvc2
- kubelet_volume_stats_health_status_abnormal, pvc3 <--
- ...
- whats this `CSIMigration` feature gate all about?
- t shirt ...
```
Up until now, Kubernetes automatically created a service account Secret when creating a Pod. That token Secret contained the credentials for accessing the API.
```
- NetworkPolicyStatus https://github.com/antrea-io/antrea/pull/1442
- https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/1959-service-lb-class-field/README.md
## Writing the KPNG kep
-
This is the live notes page for antrea.....
# Antrea LIVE Episode 26 etcd and CRDs and stuff
## THE FIRST ANT LIVE CODING CHALLENGE EVER
MAKE THIS REPRODOCE https://github.com/kubernetes/kubernetes/issues/65517
```
#!/bin/bash
#NAME=yyy
kubectl apply -f <(cat << EOF
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: foo${NAME}s.stable.example.com
spec:
group: stable.example.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
scope: Namespaced
names:
plural: foo${NAME}s
singular: foo${NAME}
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: Foo${NAME}
shortNames:
- foo${NAME}s
EOF
)
kubectl get foo${NAME}s
echo 2
kubectl get foo${NAME}s
echo 3
kubectl get foo${NAME}s
kubectl delete CustomResourceDefinition foo${NAME}s.stable.example.com
```
# Antrea LIVE Episode 24 etcd , ricardo, ...
https://github.com/ahrtr/etcd-issues/tree/master/issues/13766
```
while true ; do etcdctl --endpoints="https://localhost:2379" --cacert="/etc/kubernetes/pki/etcd/ca.crt" --cert="/etc/kubernetes/pki/etcd/server.crt" --key=/etc/kubernetes/pki/etcd/server.key endpoint status --cluster ; sleep 1 ; done
```
perf
```
etcdctl --endpoints="https://localhost:2379" --cacert="/etc/kubernetes/pki/etcd/ca.crt" --cert="/etc/kubernetes/pki/etcd/server.crt" --key=/etc/kubernetes/pki/etcd/server.key check perf --load=xl
```
# Antrea LIVE Episode 21
Introduction of antrea policy:
- what it is
- how to use it
- how to debug it
# Antrea LIVE Episode 20
## News
Antrea 1.5.1 - https://github.com/antrea-io/antrea/releases/tag/v1.5.1
- Azure - TKG CAPZ w/ Antrea default installation
- [x] https://github.com/vmware-tanzu/tanzu-framework/blob/main/pkg/v1/providers/infrastructure-azure/v1.0.1/ytt/base-template.yaml#L22
- [x] Configmap `AntreaNetworkPolicy: false`
- [x] Tests a few antrea netpol
- Antrea AKS installation
- [ ] https://antrea.io/docs/v1.5.1/docs/aks-installation/
- [ ] FULL replacement of Azure CNI -> Antrea (Netpol + CNI/IPAM)
- AKS-Engine (deprecated) can provide Netpol on Antrea and IPAM + routing on Antrea CNI plugins.
- [ ] Explore both /etc/cni/ | /opt/cni
- [ ] trafficMode=networkpolicyonly
- [ ] Looking code on Azure CNI
# Antrea LIVE Episode 19
Introduction of antrea egress and demo how to use it
# Antrea LIVE Episode 18
- Elenaor ! wants to play w/ the application path
- velero, sonobuoy, ...
- GKE external loadbalancing ?
- kubernetes pods w/ VRFs
- cant use AVI
- cant use k8s services
- ...
- VRFs vs overlays
- vrf: hardware routers etc
- vrf: not for modern apps
# Antrea LIVE Episode 17
- Lets break the CNI down W/ Mike Zappa
- Quick CNI Review (Inputs, Outputs, ENV VARS)
- Diagram of what we are going to do build
- Workflow End to End Kubectl -> Bridge/Host-Local CNI Plugin
- ADD/DEL/CHECK
- Demo's
- K8s manual setup
- basic shell cni plugin
- cnitool
- BGP
# Antrea LIVE Episode 16
### News
[Containerd 1.6 - HostProcess Container support for Windows](https://github.com/containerd/containerd/releases/tag/v1.6.0)
- [Kubernetes Policy Management Whitepaper](https://github.com/kubernetes/sig-security/blob/main/sig-security-docs/papers/policy/CNCF_Kubernetes_Policy_Management_WhitePaper_v1.pdf)
- [NetworkPolicy Status KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/2943-networkpolicy-status)
```
Conditions []metav1.Conditon
```
- antrea cni 0.3.0 -> 0.4.0 (CHECK) ?
- `kubectl delete pod -n kube-system -l "component=antrea-agent"`
- containerd -> go-cni
- runc
- containerd (go-cni) cni add
- co-cni imports. github.com/containernetworking/cni v1.0.1
- parses the cni json config
- matches 0.4.0 to the versions of its own cni config parser
## Links
https://www.cni.dev/docs/spec/
# Antrea LIVE Episode 15
### Routable Pods
- Scott Rosenberg shares about nsx integration with antrea on tkgm for routable pods
### Kubebuilder and lb operator for k8s
- Xinqi gives a brief introduction about kubebuilder and then take a look at a real operator "load balancr operator for kubernetes"
# Antrea LIVE Episode 14
### K8s news
- Aiden Obeley is from philly
- Interesting IPVS question https://kubernetes.slack.com/archives/C09QYUH5W/p1643248958272400
```
I have some questions about why kube-proxy with --proxy-mode=ipvs would always
/d a LoadBalancer Service's ExternalIP to the dummy IPVS interface (kube-ipvs0).
We noticed this while changing kube-proxy (v1.20.11) from --proxy-mode=iptables to --proxy-mode=ipvs.
When the IPVS Proxier handle LB Service, it calls syncService() and always bind the LB's ExternalIP to the kube-ipvs0 dummy interface.
Meanwhile, there doesn't seem to be similar things done in iptables Proxier.
I'd like to understand why the binding is assumed by default.
Is it meant to hairpin/SNAT traffic from the nodes that would go to the LoadBalancer?
```
- https://github.com/kubernetes/enhancements/tree/master/keps/sig-multicluster/1645-multi-cluster-services-api?utm_source=thenewstack&utm_medium=website&utm_campaign=platform
# Antrea LIVE Episode 13
### Antrea News
- Antrea v1.5 is out! [release notes](https://github.com/antrea-io/antrea/releases/tag/v1.5.0)
- Highlights of this release include the addition of multi-cluster functionalities, multicast support and improvements to Antrea IPAM
### AVI Software Load Balancer and Ingress Gateway
- What is expected out of enterprise grade k8s ingress
- Avi LB and Ingress solution
- Technical deep dive and demo
- Global load balancing for multi cluster(DNS based)
### AVI in Tanzu
- Create a vSphere cluster with ako installed using Tanzu
- Deep dive into the structure of ako-operator
- Use AVI as control plane High Availability provider
- Separate the vip network of load balancer type of service
### Audience Q's
- audience question calico vs antrea
# Antrea LIVE Episode 12
### NOTES:
- antrea 1.5 release !!! ->
- [why-we-need-lens-as-a-kubernetes-ide](https://containerjournal.com/editorial-calendar/best-of-2021/why-we-need-lens-as-a-kubernetes-ide
- [5 Kubernetes trends to watch in 2022](https://enterprisersproject.com/article/2022/1/5-kubernetes-trends-watch-2022)
- https://github.com/bentoml/BentoML
- QUESTION: How does NSX / antrea hook interoperate over OVS ?
### k8s-service-validator
- problem statement
- purpose and soluton
- CNIs
- pod-to-pod connectivity matrix
- run
- integration with sonobuoy
- run on your laptop
- code walkthrough
- how to build your test with our validator? an example: ProxyNameLabel
- unit tests
- welcome contributers
- Github: https://github.com/K8sbykeshed/k8s-service-validator
# Antrea LIVE: Episode 11
- NEWS
- [Reserve Service IP Ranges For Dynamic and Static IP Allocation](https://github.com/kubernetes/enhancements/issues/3070)
- [How to Use the Windows CSI Proxy and CSI SMB Driver for Kubernetes](https://www.phillipsj.net/posts/how-to-use-the-windows-csi-proxy-and-csi-smb-driver-for-kubernetes/)
- [Kubernetes Release 1.24 schedule](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.24)
- [Kubernetes is Moving on From Dockershim](https://kubernetes.io/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/)
- [SIG Windows DEV tools](https://github.com/kubernetes-sigs/sig-windows-dev-tools) (jay)
* [x] How it's installed
* Makefile targets
* [x] Machines bootstrap
* Vagrant provision
* image-builder vbox hypervisor [[1]](#link-1)
* support for images used downstream by ClusterAPI
- [Deep Dive Antrea CNI on Windows](https://github.com/antrea-io/antrea/blob/main/docs/windows.md) (amim)
* [ ] setup files: forked/[0-1]-antrea.ps1
* challenges in the node
* [x] OVS Windows binaries debugging (thanks [Alin](https://twitter.com/aserdean_))
* Missing DLLs [[2]](#link-2)
* `Invoke-Item ovsdb-server.exe`
* [x] Multiple NICs, Kubelet and CNI overlay configs
* [x] Windows: when `kubectl describe` isnt enough...
* Not so descriptive failure: `Unknown desc = failed to reserve sandbox name`
* Logs from `containerd` goes to ETW (event tracing for windows) - how to extract them?
* ETW output how to extract? [[3]](#link-3)
* `sc.exe config containerd binPath="c:\Program Files\containerd\containerd.exe --run-service --log-file=c:\tmp\containerd.log"`
* Ops - `failed to find plugin \"portmap\" in path [C:/opt/cni/bin]`
* Wrong (Linux) CNI configuration downloaded :)
* PS C:\etc\cni\net.d> cat .\10-antrea.conflist
- Kubernetes Services Session Affinity
* [ ] Services and Endpoints
* [ ] ClusterIP vs. NodePort
* Expectation of load balancing on both services types.
* [ ] Antrea-Proxy and services types
* ClusterIP -> Antrea-proxy
* NodePort -> Kube-proxy userspace [[4]](#link-4)
* [ ] SessionAffinity and TTL configuration
* Sticky client connection to the same pod.
* `clientIP.timeoutSeconds` - set the maximum session sticky time.
* Two odd issues on winuserspace:
1. New requests zero the expiration counter. In constant traffic flow, it never expires.
2. ttlSeconds is ignored on endpoint update. [[5]](#link-5)
* [ ] Antrea NodePortLocal [[6]](#link-6)
- Next steps
* [ ] Kube-proxy [win]userspace deprecation [[7]](#link-7)
* [ ] Antrea-proxy and Antrea Agent features!!
* [ ] Windows Operational Readiness KEP [[8]](#link-8)
* [ ] [kPNG](https://github.com/kubernetes-sigs/kpng/)
## Links
<span id="link-1">[1]</span> https://github.com/kubernetes-sigs/sig-windows-dev-tools/tree/master/experiments/image-builder
<span id="link-2">[2]</span> https://www.dependencywalker.com/
<span id="link-3">[3]</span> https://github.com/kubernetes-sigs/sig-windows-tools/pull/197
<span id="link-4">[4]</span> https://thefind.live/posts/winuserspace/readme/
<span id="link-5">[5]</span> https://github.com/kubernetes/kubernetes/pull/107399/
<span id="link-6">[6]</span> https://antrea.io/docs/main/docs/node-port-local/
<span id="link-7">[7]</span> https://github.com/kubernetes/kubernetes/issues/103860
<span id="link-8">[8]</span> https://github.com/kubernetes/enhancements/pull/2975
https://github.com/nicolaka/netshoot
### Commands
#### Disabled remote desktop firewall
`Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server'-name "fDenyTSConnections" -Value 0`
`Enable-NetFirewallRule -DisplayGroup "Remote Desktop"`
# Antrea LIVE: Episode 10
## NEWS
- how tanzu featuregates work https://github.com/vmware-tanzu/tanzu-framework/pull/1370
- Hacking tanzu by pausing reconcilation controllers
## Links
- [NSX-T 3.2 integration with Antrea](https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/administration/GUID-9197EF8A-7998-4D1B-B968-067007C56B5C.html)
- [New imgpkg version](https://github.com/vmware-tanzu/carvel-imgpkg/releases/tag/v0.24.0)
- [Helm To Carvel Conversion Tool](https://github.com/TeraSky-OSS/helm-to-carvel-conversion-tool)
- [Terraforming Tanzu](https://github.com/tanzuformers)
- [Updated TKGm Customization Repo](https://github.com/vrabbi/tkgm-customizations)
* [Added Cilium CNI](https://github.com/vrabbi/tkgm-customizations/tree/master/tkg-customizations/custom-ytt-overlays/cluster-config-changes/cilium)
* [Added Antrea + NSX-T Integration](https://github.com/vrabbi/tkgm-customizations/tree/master/tkg-customizations/custom-ytt-overlays/cluster-config-changes/antrea-nsx-integration)
* [Added Calico with Typha and BGP Configuration](https://github.com/vrabbi/tkgm-customizations/tree/master/tkg-customizations/custom-ytt-overlays/cluster-config-changes/custom-calico)
* [Added KubeProxy IPVS config](https://github.com/vrabbi/tkgm-customizations/tree/master/tkg-customizations/custom-ytt-overlays/cluster-config-changes/ipvs)
* [Added overlay to include Thanos in TKG Prometheus package](https://github.com/vrabbi/tkgm-customizations/tree/master/tkg-customizations/tkg-extension-modifications/packages/prometheus/thanos-sidecar)
## Show Notes
# Antrea LIVE: Episode 9
## NEWS
Whats it like to maintain an ingress/gateway provider? Featuring Ricardo !
- history lesson
- the eye of horus is on the dollar bill
- ingress nginx:
- first controller, ricardo steps in for alejandro
- AVI
- LIMITED gateway support in avi right now
- experimental for customers
- GSLB : *GLobal Server Loadbalancing*
- GSLB and Avi Ingress Lab https://labs.hol.vmware.com/HOL/catalogs/lab/10883
- Contour
-
- NGINX
- Envoy
- ...
# Antrea LIVE: Episode 8
- multiple clusters, support global services visited by different member clusters.
- multicast
- flexible ipam
- per namespace
- per pod <-- new + persistant IPs
- questions from KubeCon China: @vicky
- antrea vs kube-ovn ~>
- antrea vs cillium ~>
- performance improvements ~>
- new features ~>
# Antrea LIVE: Episode 7 (CAPA cniIngressRules, )
SPECIAL GUEST @sedef savas !!! AND DWAYNE!!!!
- Sedef is o.g. from the original TKG team
- https://kubernetes.io/blog/2021/12/08/dual-stack-networking-ga/
- https://github.com/kubernetes-sigs/kpng/issues/142 Readyness states
- https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/discovery/types.go
- CAPA
- declarative cluster / infra mgmt
- https://github.com/kubernetes-sigs/cluster-api-provider-aws
- https://cluster-api-aws.sigs.k8s.io/
- same mgmt cluster
- different credentials
- what are secondary cidrblocks for???
- classic Loadbalancers
- transient gateways
- hub and spoke peering
- AWS Networking Details
- NAT gateways
- private Epis
- public subnets
- Networking.V2
- VPC-peering
- Security Groups
- IAM resources and CAPA
# Antrea LIVE: Episode 6 (Antrea Agent on K3s, Windows, GKE, ... )
- luther is here again !
- k3s demo
- test networkpolicys
- https://github.com/kubernetes/enhancements/pull/2975
- https://github.com/rancher/rke2/issues/2201
- https://github.com/kubernetes-sigs/sig-windows-dev-tools/blob/master/forked/0-antrea.ps1
# Antrea LIVE: Episode 5 (Carvel and the CNI Lifecycle inside Tanzu Framework)
- api deprecations
- https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/
- https://github.com/vmware-tanzu/tanzu-framework/pull/959/
- https://github.com/vmware-tanzu/tanzu-framework/pull/959/, addon reconcilers
- How tanzu addons for CNI, etc, work
- kubetail on https://github.com/vmware-tanzu/tanzu-framework/tree/main/addons#workflow-of-tanzu-addons-manager
- https://carvel.dev/kapp-controller/docs/latest/packaging/
- PackageRepository
- PackageMetadata
- Package
- PackageInstall
- note the `cluster` field... its multicluster by default
- kubectl edit pkg cert-manager.tanzu.vmware.com.1.5.3+vmware.2-tkg.1-zshippable
- goes to custom APIServer extension
- not a CRD
- Managed by kapp controller
# Antrea LIVE: Episode 4 (AntreaProxy, 1.4, and )
- serviceAccount selectors for networkPolicies !
- egress IPs https://antrea.io/docs/v1.3.0/docs/egress/ <-- in next version of vmware tanzu (1.2.3, ipv4)
- Installing Antrea 1.4
- AntreaProxy Enablement
- https://github.com/antrea-io/antrea/pull/2632 <-- doc updates for antreaproxy
- Running Antrea w/o kube proxy - does it work ?
- VMWare Tanzu feature gates for CNI providers and the Carvel tooling
- kapp
- ytt
- kapp-controller
- BONUS ! looking at @aojeas Conntrack fix
- https://github.com/kubernetes/kubernetes/issues/105657
- https://github.com/kubernetes/kubernetes/pull/106163/files
```
root@antrea-control-plane:/# ovs-ofctl dump-flows br-int table=41
cookie=0x1040000000000,
duration=2389.591s, table=41, n_packets=0, n_bytes=0, idle_age=2389, priority=200,tcp,reg4=0x10000/0x70000,
nw_dst=10.96.85.206, <-- CLUSTER IP OF NGINX
tp_dst=80
actions=load:0x2->NXM_NX_REG4[16..18],load:0x1->NXM_NX_REG0[19],load:0x6->NXM_NX_REG7[],
group:6 <-- a definition in OVS
```
```
root@antrea-control-plane:/# ovs-ofctl dump-groups br-int 6
NXST_GROUP_DESC reply (xid=0x2):
group_id=6,
type=select,
bucket=bucket_id:0, <-- buckets?
weight:100,
actions=load:0xa0a0204->NXM_NX_REG3[],
load:0x50->NXM_NX_REG4[0..15],
resubmit(,EndpointDNAT)
```
# Antrea LIVE: Episode 3 (Multus, Whereabouts, and host-local IPAM)
- host ports on windows, do they work? or do you need nodeportlocal
- IF -> elastic IP (aws CNI) -> limited
- encap
- `service-proxy-name` <-- annotation on k8s services that disables proxying
- why multiple nics
- bypass CNI network for perf, minimize hops
- sr-iov/dpdk/edp
- multus quickstart:
- hostlocal allocation per node
- skip dchp, allocate from a pool
- fork of `static` plugin
- shim + thick component (k8s controller)
- sriov has a device plugin.
# Show Details
Show link: https://youtu.be/Q1CBFoMAG2g
Live notes: https://hackmd.io/wxNOmhZdRNm_hJzZNFKwRg
https://github.com/antrea-io/antrea/tree/main/docs/cookbooks/multus
## notes from jiunjen:
- CNF/secondary network support in Antrea in collaboration with Intel ... SR-IOV is here: https://github.com/antrea-io/antrea/pull/2651
- Native support means it (no external CNIs required)
# Episode 2
## Show Details
# Antrea LIVE: Episode 2 (antrea CNI and ovs metrics + a new sessionAffinity kubeproxy bug)
# Show Details
Show link: https://www.youtube.com/watch?v=3aUnws6diAY
## Antrea stuff
- vivek, multus, and multus+telco networking
- https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md
- Connection tracking (“conntrack”) is a core feature of the Linux kernel's networking stack. It allows the kernel to keep track of all logical network connections or flows, and thereby identify all of the packets which make up each flow so they can be handled consistently together
- conntrack is leveraged by stateful services, like stateful firewall, NAT, that require to track connections. The connections are not just TCP, and can be UDP, ICMP, SCTP, etc.
- Flow record, 2 records per connection, aggregate
- ELK Stack super cool; Antrea layer 4 service mesh
- future: policy recommendation engines + performance analytics
- prometheus installation and OVS monitoring
```
$> kubectl apply -f build/yamls/antrea-prometheus.yml
$> sleep 60 ; kubectl port-forward pod/prometheus-deployment-79fb7d997f-nkbjj -n monitoring --address 0.0.0.0 9090:9090 -n monitoring
```
- table 90
- https://github.com/antrea-io/antrea/blob/main/docs/design/ovs-pipeline.md
- https://github.com/antrea-io/antrea/blob/main/docs/prometheus-integration.md
## New in upstream
- a new kube-proxy affinity bug https://github.com/kubernetes/kubernetes/pull/106030
- https://github.com/kubernetes/kubernetes/issues/105657
- https://github.com/K8sbykeshed/k8s-service-lb-validator/issues/35
----
# Episode 1
## Show Details
https://www.youtube.com/watch?v=aWUwxQ58bEQ
### Antrea 1.3.0
- kubectl apply -f https://github.com/antrea-io/antrea/releases/download/v1.3.0/antrea.yml
- install ~ jayunit100/k8sprototypes/kind/ kind-local-up.sh
### FQDN Policies
```
apiVersion: crd.antrea.io/v1alpha1
kind: ClusterNetworkPolicy
metadata:
name: acnp-fqdn-all-foobar
spec:
priority: 1
appliedTo:
- podSelector:
matchLabels:
app: client
egress:
- action: Drop
to:
- fqdn: "*foobar.com"
```
## K8sNetLook
- https://github.com/sarun87/k8snetlook
## Notes
- Release Notes: https://github.com/antrea-io/antrea/releases/tag/v1.3.0
- https://github.com/kubernetes/enhancements/pull/2522
- `kubectl get acnp`
- `CLUSTER=antrea CONFIG=kind-conf.yaml ./kind-local-up.sh`
- https://kubernetes.io/docs/concepts/services-networking/network-policies/
- https://github.com/sarun87/k8snetlook/blob/master/examples/run-k8s.yaml
- bitnami/nginx image to avoid docker limits
- Arun's awesome kubecon talk slides: https://kccncna2021.sched.com/event/lV0b