# DevOps Dilemmas (voda)- 2023-09 # What is DevOps > DevOps is a set of practices, principles, and cultural philosophies that aim to improve collaboration between **dev**elopmetand IT **op**erations teams. > goals: - automate software delivery processes - automate infrastructure management processes - enable faster and more reliable software development and deployment - break down silos between dev and ops, promote CI/CD # DevOps dilemmas - Automation Complexity - Cultural Shift - Monitoring and Observability - Tooling and Technology Stack [https://coub.com/view/2cqwtn](https://coub.com/view/2cqwtn) - Scalability and Resource Management # Monitoring - 4 golden signals -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled.png) # Observability - Rise of Microservices ~ 90% of enterprise is adopting Microservices architecture - Metrics (what) - Logs (why) - Traces (where) -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%201.png) # What is GitOps > GitOps is a modern approach to managing and automating the deployment and operation of software systems and infrastructure using Git as the source of truth > - Declarative Configuration - Continuous Deployment - Pull-Based Model - Reconciliation Loop - Auditing and Rollback # Declarative Configuration > The desired state of the system is defined in declarative configuration files stored in the Git repository. These files describe how the infrastructure and applications should look and behave. > - App: Kubernetes: manifest yaml / helm - Infra: - CloudProvider specific - AWS Cloudformation (yaml / json) - Google Cloud Deployment Manager (yaml + python) - Azure resource Manager Templates (json) - All-in-k8s: same as app - Terraform (HCL,json) - new license ??? - Pulumi (js,python,go code) - Oldschool: Chef/Puppet/Salt/Ansible # CI/CD meaning - **Continuous Integration:** Frequently merging into the main codebase - **Continuous Delivery:** Code **can** be pushed to production at any time - **Continuous Deployment:** Code **is p**ushed to production automatically # GITOPS Components - Git repository - CI/CD pipeline - GitOps agent - Kubernetes cluster # Desired state -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%202.png) # Gitops Workflow 1. Developer commits updates to Git repo for app 2. Commit triggers a CI pipeline (build, test, etc) 3. New container image pushed to registry 4. Webhook triggered to update deployment config with new app image (or, GitOps agent performs this task) 5. GitOps agent reconciles changes in live system # Workflow diagram -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%203.png) # Workflow Evolution -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%204.png) # Demo Workflow -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%205.png) # Batteries to choose from 1. Push vs. Pull Deployments 2. Repository Structure 3. Secrets Management 4. Change Management 5. Updates to Config repository # **Push vs. Pull Deployments** - **Push -** Tooling 'external' to cluster pushes changes to the cluster using kubectl/helm - **Pull -** Agent inside the cluster monitors repo and pulls and applies changes -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%206.png) # Push Based Deployment **Pros**: - Familiar tools - Extends existing pipelines **Cons:** - Difficult to measure and respond to state gap - Credentials outside cluster boundary -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%207.png) # **Pull-based Deployments** Agent in cluster continuously tracks desired state defined in Git repo, and applies any changes to the cluster state. **Pros:** - Can limit direct access to cluster - Continuously monitors and reports on delta between desired and actual state **Cons:** - Limited choice of tools available - Handling secrets can be troublesome # Pull Diagram -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%208.png) # Repository structure - per namspace - per team - per cluster -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%209.png) -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%2010.png) -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%2011.png) # Secret Management - TechRadar See full report: [https://radar.cncf.io/2021-02-secrets-management](https://radar.cncf.io/2021-02-secrets-management) -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%2012.png) # Secret Management - Centralized Use a centralised secret management system (e.g. AKS/GKS/Vault) **pros:** - Tried and tested ‘enterprise’ grade solution **cons:** - System and credentials exists outside of the GitOps process - Heavyweight / Complex solution # Secret Management - Sealed Secrets Github Repo: [https://github.com/bitnami-labs/sealed-secrets](https://github.com/bitnami-labs/sealed-secrets) Example: [https://github.com/lalyos/sealed-secrets-demo](https://github.com/lalyos/sealed-secrets-demo) -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%2013.png) # Change Management The main feature of GitOps: - Git repo over which you can manage Pull Requests - Team members can propose changes, whilst others can merge those changes. - Detailed history with audit trail # Updates to Config Repo - If we want to have automated deployments then we don't want to be editing our config repository by hand. - Our CI pipeline has to push changes to it automatically. - How can we update it from CI? # Updates to Config Repo - Scripts in CI pipeline Implement a script at the end of the CI Pipeline **pros:** - Easy solution to implement - Uses existing tooling **cons:** - Could result in conflicts due to race conditions # Updates to Config Repo - **Watch image registry** Watch the image registry for changes. The CI Pipeline pushes an image to registry rather updating YAML manifest. **pros:** - Removes need for CI to update config in repo **cons:** - Not yet a universal GitOps solution - Trust operator to make changes to repo # Gitops Agents [https://landscape.cncf.io/card-mode?category=continuous-integration-delivery&grouping=category](https://landscape.cncf.io/card-mode?category=continuous-integration-delivery&grouping=category) -%202023-09%20b6d11f90981a4488b57be45dc2fe6eab/Untitled%2014.png)