Windows node support

--- kep-number: 0 title: Windows node support authors: - "@benmoss" owning-sig: sig-windows participating-sigs: - sig-architecture reviewers: - TBD approvers: - TBD editor: TBD creation-date: 2019-11-29 last-updated: 2019-11-29 status: provisional --- # Windows node support ## Table of Contents * [Windows node support](#windows-node-support) * [Table of Contents](#table-of-contents) * [Summary](#summary) * [Motivation](#motivation) * [Goals](#goals) * [Non-Goals](#non-goals) * [Proposal](#proposal) * [Risks and Mitigations](#risks-and-mitigations) * [Graduation Criteria](#graduation-criteria) * [Implementation History](#implementation-history) ## Summary There is strong interest in the community for adding support for workloads running on Microsoft Windows. This is non-trivial due to the significant differences in the implementation of Windows from the Linux-based OSes that have so far been supported by Kubernetes. ## Motivation Windows-native workloads still account for a significant portion of the enterprise software space. While containerization technologies emerged first in the UNIX ecosystem, Microsoft has made investments in recent years to enable support for containers in its Windows OS. As users of Windows increasingly turn to containers as the preferred abstraction for running software, the Kubernetes ecosystem stands to benefit by becoming a cross-platform cluster manager. ### Goals - Enable users to run nodes on Windows servers - Document the differences and limitations compared to Linux - Test results added to testgrid to prevent regression of functionality ### Non-Goals - Adding Windows support to all projects in the Kubernetes ecosystem (Cluster Lifecycle, etc) ## Proposal As of 29-11-2018 much of the work for enabling Windows nodes has already been completed. Both `kubelet` and `kube-proxy` have been adapted to work on Windows Server, and so the first goal of this KEP is largely already complete. ### What works today - Windows-based containers can be created by kubelet, [provided the host OS version matches the container base image](https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility) - ConfigMap, Secrets: as environment variables or volumes - Resource limits - Pod & container metrics - Pod networking with [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md), [OVN-Kubernetes](https://github.com/openvswitch/ovn-kubernetes), [two CNI meta-plugins](https://github.com/containernetworking/plugins), [Flannel](https://github.com/coreos/flannel) and [Calico](https://github.com/projectcalico/calico) - Dockershim CRI - Many<sup id="a1">[1]</sup> of the e2e conformance tests when run with [alternate Windows-based images](https://hub.docker.com/r/e2eteam/) which are being moved to [kubernetes-sigs/windows-testing](https://www.github.com/kubernetes-sigs/windows-testing) - Persistent storage: FlexVolume with [SMB + iSCSI](https://github.com/Microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows), and in-tree AzureFile and AzureDisk providers <sup id="a1">1</sup> This list should be available at https://k8s-testgrid.appspot.com/sig-windows but this test setup is not currently working. https://k8s-testgrid.appspot.com/google-windows#windows-prototype is also running against a Windows cluster. ### What will work eventually - `kubectl port-forward` hasn't been implemented due to lack of an `nsenter` equivalent to run a process inside a network namespace. - CRIs other than Dockershim: CRI-containerd support is forthcoming ### What will never work (without underlying OS changes) - Certain Pod functionality - Privileged containers - Reservations are not enforced by the OS, but overprovisioning could be blocked with `--enforce-node-allocatable=pods` (pending: tests needed) - CSI plugins, which require privileged containers - [Some parts of the V1 API](https://github.com/kubernetes/kubernetes/issues/70604) - Overlay networking support in Windows Server 1803 is not fully functional using the `win-overlay` CNI plugin. Specifically service IPs do not work on Windows nodes. This is currently specific to `win-overlay` - other CNI plugins (OVS, AzureCNI) work. ### Relevant resources/conversations - [sig-architecture thread](https://groups.google.com/forum/#!topic/kubernetes-sig-architecture/G2zKJ7QK22E) - [cncf-k8s-conformance thread](https://lists.cncf.io/g/cncf-k8s-conformance/topic/windows_conformance_tests/27913232) - [kubernetes/enhancements proposal](https://github.com/kubernetes/features/issues/116) ### Risks and Mitigations **Second class support**: Kubernetes contributors are likely to be thinking of Linux-based solutions to problems, as Linux remains the primary OS supported. Keeping Windows support working will be an ongoing burden potentially limiting the pace of development. **User experience**: Users today will need to use some combination of taints and node selectors in order to keep Linux and Windows workloads separated. In the best case this imposes a burden only on Windows users, but this is still less than ideal. ## Graduation Criteria ## Implementation History ## Other references [Past release proposal for v1.12/13](https://docs.google.com/document/d/1YkLZIYYLMQhxdI2esN5PuTkhQHhO0joNvnbHpW68yg8/edit#)