# Office Hours for April 2020 ## West Coast Edition ### Panelists - Jorge Castro, VMware (Host) - Samudrala Vamshi, American Airlines - Monica Rodriguez, VMware - Tunde Olu-Isa, VMware - Erik Osterman, Cloud Posse - Jeremy Rickard, VMware - Dave Strebel, Microsoft - Bhargav Bhikkaji(BB), Independent Consultant. 1. Question: - Text : Rotating certs - Asker: Panel answering Quetions from previous session - Answer: https://github.com/kubernetes/website/pull/19351 Ansible playbooks 2. Question: - Text: Do you have any suggestions on how to setup prometheus for cluster monitoring to keep it highly available? ie, would you put it in the same cluster, a separate cluster or an external vm? Do you have any resources on how to setup a "master" prometheus? I am not sure how to send from one prometheus to another. - Asker: Bre Gielissen - Answer: Prometheus inside cluster ( issues with memory clog) Send to another one outside cluster Open source projects - storage backend Helm chart installation prometheus docs telegraf + influx for kubernetes monitoring is also a good choice trickster- https://github.com/tricksterproxy/trickster https://prometheus.io/docs/prometheus/latest/federation/ https://github.com/thanos-io/thanos https://www.instana.com/ https://istio.io/docs/tasks/observability/kiali/ 3. Question: - Text - How to use Kafka more efficiently with K8s? Was using sync communication with Kafka from Microservice, but when it hits the FD limit, Microservice goes into restart mode. We used async communication later, but what are other ways to use it more efficiently? - Asker: WarDaddy - Answer: -https://github.com/danielqsj/kafka_exporter and https://github.com/lightbend/kafka-lag-exporter https://stackoverflow.com/questions/33649192/how-do-i-set-ulimit-for-containers-in-kubernetes 4. Question: - Text What are folks doing for inventory / tracking if running many clusters with many nodes? Any slick inventory operators or ways to track clusters? - Asker: jimangel - Answer: https://infra.app/ https://kubecost.com/ https://github.com/kubectl-plus/kcf https://github.com/rchakode/kube-opex-analytics https://rancher.com/blog/2020/fleet-management-kubernetes/ https://github.com/getsentry/sentry https://github.com/lensapp/lens 4. Question: - Text Hello.. Is there a way to get notified if Kubernetes pod goes out of memory without using kubestate metrics? (Basically the OOM killed terminated reason) - Asker: Saloni - Answer: -https://www.amazon.com/Kubernetes-Best-Practices-Blueprints-Applications-ebook/dp/B081J62KLW 5. Question: - Text: What resources are people using to help developers and junior SREs get up the steep learning curve of K8s?Coming from a mix of backgrounds using ecs, docker-compose, heroku, terraform, puppet, and some bare metal. We have some passionate k8s advocates and some who don't know where to begin. - Asker: Aaron Eaton - Answer: - https://www.katacoda.com/learn - https://kind.sigs.k8s.io/docs/user/quick-start/ - https://kubernetes.io/docs/tasks/tools/install-minikube/ - https://github.com/kelseyhightower/kubernetes-the-hard-way https://azure.microsoft.com/en-in/resources/kubernetes-up-and-running/ https://kubernetes.io/ https://kubernetes.io/training/ https://training.linuxfoundation.org/training/kubernetes-for-developers/#outline https://www.edx.org/course/introduction-to-kubernetes https://www.edx.org/course/introduction-to-linux https://killer.sh/course/preview/e84d0e31-4fff-4c42-8afd-be1bdbc0d994 https://github.com/dgkanatsios/CKAD-exercises/blob/master/a.core_concepts.md https://kubernetes.io/docs/reference/kubectl/cheatsheet/ https://learnk8s.io/troubleshooting-deployments https://learnk8s.io/academy https://kodekloud.com/ https://kubernauts.de/en/training/kubernetes-training-course.html https://docs.google.com/presentation/d/13EQKZSQDounPC1I6EC4PmqaRmdCrpT3qswQJz9KRCyE/edit#slide=id.g33599df588_13_58 https://collabnix.github.io/kubelabs/ https://www.katacoda.com/courses/kubernetes/playground Pluralsight is free till 30th April https://eksworkshop.com/ 6. Question: - Text: I am currently setting up a bash script to setup our dev/staging GKE cluster so we can shut them down at night and restart them on mornings at will, but Im wondering if there's any solution that exists that would allow me to bundle up all the resources needed, the cluster, the dns, the various public help charts (mongo, redis, etc..) with custom config and our own applications (happens to be also in custom helm charts) rather than a custom bash script? I know AWS has something called CloudFormation that might have allowed me to do this but we are using GKE. If many tools exists, which one you recommend? - Asker: Christian Roy - Answer: Terraform ,flux, Argo CD https://github.com/cloudposse/terraform-aws-eks-cluster https://eksctl.io/ 7. Question: - Text:earlier today, I was troubleshooting an issue and noticed that in a cluster there are deployment using pvc with a storage class local-storage and others with local-storage-local, however there is SC object when I do "get sc" that presents the storage class, how did that happen? also tried to check the node for annotation/labels related, there was one label localstorage=true but nothing corresponding in the deployment - Asker: Walid 8. Question: - Text: we are running machine learning model in kubernetes pod.if my pod is reached 100% pod resources automatically restarted but need to make some delay 10 mits to restart.how we can do? - Asker: devopsdymyr - Answer : set resource limits https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/ , maybe pod prestop can help 9. Question: - Text: Do you have any suggestions on how to organize the yaml files for kubernetes resources? Right now, our cluster was setup with a bunch of yaml files that were applied manually which is very difficult to track. Would you use helm charts heavily there or something like a gitops workflow? - Asker: Bre Gielissen - Answer : Helm charts, Kustomize https://cloudposse.com/change-management/ https://github.com/roboll/helmfile https://www.openpolicyagent.org/docs/v0.12.2/kubernetes-admission-control/ 10. Question: - Text: What’s the purpose of setting a CPU limit if the container might be allowed to exceed it and won’t be killed or is it just to override the Limit Range admission controller? How does k8s knows whether if it will allow or not, does it depends on the allocatable resources of the node? - Asker: Javier - Answer: Its a cgroup behavior -https://github.com/kubernetes-sigs/descheduler 11. Question: - Text:What’s the best persistent storage for on-premise workloads? I have tried rook-ceph and Longhorn. What are you guys using? - Asker: Jawiz - Answer: rook, minio, portworx, NFS, and investigating vsphere(VSan) CSI - as a database geek, I do local volumes when I'm performance-constrained (and rely on DB replication), and when I'm not I do Ceph/Rook URL's: - CloudPosse:https://cloudposse.com/ - Rotating certs: https://github.com/kubernetes/website/pull/19351 - Prometheus: - https://prometheus.io/docs/prometheus/latest/federation/ - https://github.com/thanos-io/thanos - https://medium.com/faun/comparing-thanos-to-victoriametrics-cluster-b193bea1683 - https://github.com/tricksterproxy/trickster - https://www.instana.com/ - https://istio.io/docs/tasks/observability/kiali/ - Kafka: - https://github.com/danielqsj/kafka_exporter - https://github.com/lightbend/kafka-lag-exporter - https://stackoverflow.com/questions/33649192/how-do-i-set-ulimit-for-containers-in-kubernetes - General Url's for prepartion - https://infra.app/ - https://kubecost.com/ - https://github.com/kubectl-plus/kcf - https://github.com/rchakode/kube-opex-analytics - https://rancher.com/blog/2020/fleet-management-kubernetes/ - https://github.com/getsentry/sentry - https://github.com/lensapp/lens - https://www.amazon.com/Kubernetes-Best-Practices-Blueprints-Applications-ebook/dp/B081J62KLW - https://www.katacoda.com/learn - https://kind.sigs.k8s.io/docs/user/quick-start/ - https://kubernetes.io/docs/tasks/tools/install-minikube/ - https://github.com/kelseyhightower/kubernetes-the-hard-way - https://azure.microsoft.com/en-in/resources/kubernetes-up-and-running/ - https://kubernetes.io/ - https://kubernetes.io/training/ - https://training.linuxfoundation.org/training/kubernetes-for-developers/#outline - https://www.edx.org/course/introduction-to-kubernetes - https://www.edx.org/course/introduction-to-linux - https://killer.sh/course/preview/e84d0e31-4fff-4c42-8afd-be1bdbc0d994 - https://github.com/dgkanatsios/CKAD-exercises/blob/master/a.core_concepts.md - https://kubernetes.io/docs/reference/kubectl/cheatsheet/ - https://learnk8s.io/troubleshooting-deployments - https://learnk8s.io/academy - https://kodekloud.com/ - https://kubernauts.de/en/training/kubernetes-training-course.html - https://docs.google.com/presentation/d/13EQKZSQDounPC1I6EC4PmqaRmdCrpT3qswQJz9KRCyE/edit#slide=id.g33599df588_13_58 - https://collabnix.github.io/kubelabs/ - https://www.katacoda.com/courses/kubernetes/playground - https://eksworkshop.com/ - https://github.com/cloudposse/terraform-aws-eks-cluster - https://eksctl.io - https://github.com/kubernetes-sigs/descheduler - https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/ - https://cloudposse.com/change-management/ - https://github.com/roboll/helmfile - https://www.openpolicyagent.org/docs/v0.12.2/kubernetes-admission-control/ ## EU Edition ### Panelists - Jorge Castro, VMware (Host) - Jeffrey Sica, Red Hat - Pierre Humberdroz, Spectrm - Puja Abbassi, Giant Swarm - Chris Carty, Google - Dave Strebel, Microsoft - Mario Loria, StockX - Povilas Versockas, UW - Yourname, Affiliation ### Links https://github.com/tekliner/rabbitmq-operator https://github.com/indeedeng/rabbitmq-operator https://kubernetes.io/docs/tasks/run-application/run-replicated-stateful-application/ https://github.com/influxdata/telegraf/issues/6959#issuecomment-608318315 https://relnotes.k8s.io/?markdown=balancer&releaseVersions=1.18.0 https://stackoverflow.com/questions/61113732/how-to-reuse-k8s-validation-in-my-own-crd-controller https://engineering.dollarshaveclub.com/kubernetes-fixing-delayed-service-endpoint-updates-fd4d0a31852c https://github.com/kubernetes/kubernetes/blob/master/go.mod#L124 https://kubernetes.io/docs/tasks/tls/certificate-rotation/ https://github.com/kubernetes/community/tree/master/sig-cluster-lifecycle https://github.com/kubernetes/kubernetes/issues/88553 https://github.com/lensapp/lens ### Questions 1. Question: - Recently I've been playing with a 3-node RabbitMQ cluster. The cluster is deployed as a StatefulSet with replica=3. For some reason, one of the pods refused to join the cluster; this led to readyness, liveness and entrypoints to fail, so one of the pods refused to start. According to RabbitMQ docs, the way to solve this is to issue a "reset" command, but how could I do that if the pod refused to run? I also thought of changing the entrypoint or the readyness probes, but this would have impacted the other pods since they are replicas (removing the readyness probe could prevent them to start in the right order for example). - I ended up resetting the RabbitMQ cluster state in a somewhat-less-orthodox way, but this is not something I could have done in a production environment; I wondered if I could have recovered it in some other way (possibly less application-specific) like for example, if I could get a command to run in a pod that has problems on starting, or if I could override the readyness/liveness/entrypoint for a specific replica, or something else entirely? - Asker: Simone Baracchi 1. Question: - https://kubernetes.slack.com/archives/C6RFQ3T5H/p1585908749010200 - Asker: Jmorcar - Answer in here: https://github.com/influxdata/telegraf/issues/6959 1. Question: - I have an external netty server (not in cluster). How can I share same websocket connection from my pods(clients) to this external server? - I need multiple pods so that messages sent by the service can be distributed but don't want service to have separate connection for each pod. - Asker: prashant rathore 1. Question: - https://kubernetes.slack.com/archives/C6RFQ3T5H/p1585063644252200 - Asker: felixdpg 1. Question: - Hello all Since kubernetes v1.18 was released I encountered a lot of problems with MetalLoadBlancer External IP. All the services of type LoadBalancer doesn't get the address, it remain in "pending" state. I've checked the metallb logs and I didn't find anything that appeared meaningful for the problem. I'm working on a virtual environment deployed automatically, so I'm 100% sure that nothing changed in the configs, the only things changed is the kubernetes version. I've tried to deploy the cluster from scratch many times but the result was the same. I tried a kubernetes manual installation too, but the result is the same. I've used this configuration for 3 months without any problem, so I know that before kube 1.18 all worked as expected. Did you have any idea? Did you experienced a similar issue? - Asker: Emanuel 1. Question: - https://stackoverflow.com/questions/61113732/how-to-reuse-k8s-validation-in-my-own-crd-controller - Asker: mingmin 1. Question: - Hi All, I am getting the following errors from the APIServer logs does any one have any idea what these errors and how to fix. `{code="500",component="apiserver",contentType="application/json",endpoint="https",instance="172.31.111.155:443",job="apiserver",namespace="default",resource="events",scope="namespace",service="kubernetes",verb="POST",version="v1"}` Error from apiserver logs: `I0415 12:47:08.913951 1 log.go:172] http: TLS handshake error from 172.31.111.240:41740: remote error: tls: bad certificate` - Asker: Ravi 1. Question: - Why can't the kubedm certificate be valid for ten or one hundred years? It's too painful. When we deliver kubernetes to the customer's production environment, we need to manually compile the kubedm source code or manually create a large number of certificates. Can't you give us an optional parameter or variable to flexibly configure this option - Asker: zhang 1. Question: - I have a scenario where I see sometimes the endpoints do not gets created even when the pod is running service is there and labels are matching, I did see one blog as well https://engineering.dollarshaveclub.com/kubernetes-fixing-delayed-service-endpoint-updates-fd4d0a31852c but it didnt help, finally eneded up writing my own python script to handle this scenario. Where when you delete and recreate the service it immediately gets created. But this should nit happen ideally any thoughts on this? - Asker: sammy 1. Question: - Just wondering why kubernetes is using make and bazel I made a simple PR for an error handling and I had several CI stages failed because of bazel - Asker: Rodrigo Villablanca 1. Question: - Text - Asker: 1. Question: - Text - Asker:
Last changed by  
 
Kubernetes
Production grade container orchestration, made with love
1414

Read more

Lessons Learned from 3 years of Kubernetes Office Hours

Date: 23 August 2020 Authors: Jorge Castro (VMware), Josh Berkus (Red Hat), Chris Carty (Google), Dan "POP" Papandrea (Sysdig), David McKay (Equinix Metal) Today we celebrate three years of the Kubernetes Office Hours. This is a monthly event where we take a panel of volunteers, stick them on a live stream, and then see how many questions we can field from the community. I started the show for one reason. I had recently started my Kubernetes journey and was learning all these new concepts while rewiring my traditional sysadmin brain to be more cloud native. So the idea was if I'm going to dig into this stuff and bother my new coworkers with silly questions, we might as well do it together and on the air to share our experiences and do it in a way that is fun and useful for others. Give away some tshirts, fame and fortune would surely follow. After 65 episodes we've decided to take a look at some of the more common problem areas that we've been tackling, and put together a quick summary for you on things where you might want to invest your attention. You will find many articles on "Top X things to know about Kubernetes". We've specifically avoided those and went back into our archives because what people think you need to know and what you actually need to know can be different.

3/18/2021
Office Hours Feb 2021

EU Edition Panelists Person: Andrew Question: We're writing a controller with controller-runtime, and trying to use the Generation/ObservedGeneration pattern to avoid reconciling if there isn't any change (not using the predicate provided by controller-runtime for that purpose yet though). My question is how can that work with the possibility of a stale cache? When we write the ObservedGeneration to the Status of our CR, it triggers another reconcile immediately, but in some cases, the cache is stale and the CR it "Get"s still has the old Status, and therefore the old ObservedGeneration. What is the recommended strategy of dealing with this? Thanks! Person: Simone Baracchi Question: I'd like to configure my small cluster as "highly available" with no single master / single point of failure and make the best use of all the cluster resources. My current plan is to make 3 nodes run as masters and be able to schedule pods on the masters. From my research the issues in doing so are 1) security issues about sensitive data on master which could be read from malicious pods and 2) pods competing for resources (especially in case of a node failure). I'm not too concerned about security atm, and I can think of limiting the max number of pods / resources used. Is there any other red flag in doing so? Person: Jesper Berg Axelsen

2/17/2021
Contributors Awards Ceremony Playbook

awards/ README.md -- list of everyone playbook.md Kubernetes has a tradition of yearly peer awards. These are commonly handed out during the end-of-year Contributor Summit. The CNCF typically has awards as part of KubeCon + CloudNativeCon, the Kubernetes awards are a compliment to those, and are meant to be peer awards. Ideals and Values Typically each SIG will nominate two(2) recipients for an award.

2/12/2021
Office Hours January 2021

EU Edition Panelists Rachel Leekin, Chris Carty, Dan POP Papandrea, Saiyam Pathak Person: Mostafa Elmenbawy (https://kubernetes.slack.com/archives/C6RFQ3T5H/p1609991530274100?thread_ts=1607960423.257700&cid=C6RFQ3T5H) Question: What is recommended for on premise production cluster spanning multiple hosts? Answer:

1/20/2021
Read more from Kubernetes
Published on HackMD