HackMDWindows node support
Table of Contents
Summary
There is strong interest in the community for adding support for workloads running on Microsoft Windows. This is non-trivial due to the significant differences in the implementation of Windows from the Linux-based OSes that have so far been supported by Kubernetes.
Motivation
Windows-native workloads still account for a significant portion of the enterprise software space. While containerization technologies emerged first in the UNIX ecosystem, Microsoft has made investments in recent years to enable support for containers in its Windows OS. As users of Windows increasingly turn to containers as the preferred abstraction for running software, the Kubernetes ecosystem stands to benefit by becoming a cross-platform cluster manager.
Goals
- Enable users to run nodes on Windows servers
- Document the differences and limitations compared to Linux
- Test results added to testgrid to prevent regression of functionality
Non-Goals
- Adding Windows support to all projects in the Kubernetes ecosystem (Cluster Lifecycle, etc)
Proposal
As of 29-11-2018 much of the work for enabling Windows nodes has already been completed. Both kubelet and kube-proxy have been adapted to work on Windows Server, and so the first goal of this KEP is largely already complete.
What works today
- Windows-based containers can be created by kubelet, provided the host OS version matches the container base image
- ConfigMap, Secrets: as environment variables or volumes
- Resource limits
- Pod & container metrics
- Pod networking with Azure-CNI, OVN-Kubernetes, two CNI meta-plugins, Flannel and Calico
- Dockershim CRI
- Many[1] of the e2e conformance tests when run with alternate Windows-based images which are being moved to kubernetes-sigs/windows-testing
- Persistent storage: FlexVolume with SMB + iSCSI, and in-tree AzureFile and AzureDisk providers
1 This list should be available at https://k8s-testgrid.appspot.com/sig-windows but this test setup is not currently working. https://k8s-testgrid.appspot.com/google-windows#windows-prototype is also running against a Windows cluster.
What will work eventually
kubectl port-forwardhasn't been implemented due to lack of annsenterequivalent to run a process inside a network namespace.- CRIs other than Dockershim: CRI-containerd support is forthcoming
What will never work (without underlying OS changes)
- Certain Pod functionality
- Privileged containers
- Reservations are not enforced by the OS, but overprovisioning could be blocked with
--enforce-node-allocatable=pods(pending: tests needed)
- CSI plugins, which require privileged containers
- Some parts of the V1 API
- Overlay networking support in Windows Server 1803 is not fully functional using the
win-overlayCNI plugin. Specifically service IPs do not work on Windows nodes. This is currently specific towin-overlay- other CNI plugins (OVS, AzureCNI) work.
Relevant resources/conversations
Risks and Mitigations
Second class support: Kubernetes contributors are likely to be thinking of Linux-based solutions to problems, as Linux remains the primary OS supported. Keeping Windows support working will be an ongoing burden potentially limiting the pace of development.
User experience: Users today will need to use some combination of taints and node selectors in order to keep Linux and Windows workloads separated. In the best case this imposes a burden only on Windows users, but this is still less than ideal.
