DevConf.US 2020 Talk: Using containers and VMs on free public CIs

# DevConf.US 2020 Talk: Using containers and VMs on free public CIs https://docs.google.com/presentation/d/1ExB6WS1ot3yhCZ-tnS2zrW4Jr1hSH6YowAXDEEyf23c/edit#slide=id.p ### Title *Limit 50 chars* Using containers and VMs on free public CIs ### Type **Presentation** Workshop Discussion ### Themes **Application Development with Serverless & Containerization** **Ensuring Software Quality** Evolving Technology Machine Learning & Artifical Intelligence Open Source & Process Operating at Scale Other theme not listed Security, Privacy & Data Governance Systems Engineering & Hardware User Experience in Open Source ### Difficulty Beginner **Intermediate** Advanced ### Abstract *Limit 1000 characters* At first glance, free public CIs seem insufficient for testing many open source projects. They provide only a single virtual machine with no choice of Linux distro, and no container or virtual infrastructure. However, many projects are publishing as containers, or even kubernetes operators. Many have conventional installers that must be tested across numerous Linux distros. Some need SELinux or other CI-disabled kernel features. This talk will explain how our project overcame all these limitations: container infrastructure and full virtual machines running on top of free public CIs, and alternative solutions that also work. The solutions will consist of a comparison of CI-compatible open source container infrastructures and hypervisors, CI design patterns for them, and other tips & tricks for integrating them. ### Time Limit 30 min. They prefer a little bit under. ### Outline * Problem statement: You cannot run VMs on CI environments, and containers must use the CI's few resources. (D) * Why would you want to use containers? (D) ** You need tools like dnf which aren't available on all distros ** Isolated environment (pip virtualenv insufficient) * Why do you need VMs sometimes? (D) ** SELinux ** FIPS * Why do you need VMs sometimes (cont'd)? ** Any other kernel features that occasionally matter ** Open source projects are supposed to support and integrate with as many distros as possible * High-level Compare of hardware VMs vs generic-hook containers * More detailed comparison of: hardware VMs - software VMs - software emulation - special-hook containers - generic-hook containers * What are free CI envs running on? With what specs? (D) * Results testing hardware KVM, software vbox and software qemu on GHA and Travis ** Email sent months ago. This should be a blog post. ** Quickly reevaluate GHA on Ubuntu 20.04. May now support hardware kvm (new azure VMs may support nested hardware virt) (M) * Pulp running hardware kvm in CI ** Problem: We haven't started doing this yet, we are not following anticipated schedule. ** Further prototyping for pulplift may be feasible. * Pulp running containers and kubernetes in CI ** Tips on running lightweight: Use k3s. Pre-build images with cron jobs. ** Problem: We haven't optimized this too heavily. We haven't explored other tips. * Design patterns: ** Problem: We haven't started running hardware KVM on CI yet. ** Presumably a separate layer to use your regular CI code. ** Integrating k3s better by using docker or other ### Misc research ## GHA (Ubuntu 20.04) hyperv (Azure) VM No nested HVM 2 CPU cores reported at 2095 Mhz (rather than 2.6 Ghz) from a Xeon 8171M 6927 MB RAM 4095 MB SWAP / : 37G free out of 84G / IOPS: 6267 / 2094 /mnt: 9GB free out of 14G - used for the 4GB swap file /mnt IOPS: 6117 / 2044 (note: iops with fio. Thinkpad X1C6 iops: 55.5k/18.5k ) IOPS benchmark: `fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75` `7z b -mmt2` total ratings: 5994, 5978, 5899 sudo fallocate -l 9G /mnt/swapfile_2 sudo chmod 600 /mnt/swapfile_2 sudo mkswap /mnt/swapfile_2 sudo swapon /mnt/swapfile_2 ## Travis (Ubuntu 20.04) 2 CPU cores reported at 2800 Mhz. Undefined Xeon. 7961 MB RAM 0 MB RAM / : 55G free out out of 68G / IOPS: 1694 / 566 `7z b -mmt2` total ratings: 4981, 4987, 5024 ## Travis lxd container HVM supported! ### Brian Profitt Advise 1. An overview slide, or more of an explanation of what you'll be talking about. Like "advice for open source projects in general, based on our experiences on Pulp." "We'll cover the solutions out there, and how to optimize them" 2. Hand-Off 3. "The problems with free, public CI environments" should have an bulleted points. 4. "Using containers" should have each bullet in a consistent format. And clarify if these are pros or cons. 5. "Why do you need VMs sometimes?" 6. "VMs vs Containers" - Bigger fonts if physical 7. He liked the Plan 9 joke 8. Split up the VirtualBox SW Mode vs Xen Slide 9. You can increase your time by grouping Xen and OpenVZ together. 10. Demo it, such as a sped up attached video. Overall, he thought the presentation was excellent. (He came from the oVirt community.) A lighter touch on the plans that don't work. Remember cues for transitions.