# Kubernetes as Code with Pulumi **Posted on Friday, Aug 11, 2023** 🤓 **Scott Rigby** If you’ve ever worked with Kubernetes (K8s), you know that beyond imperatively using the `kubectl` command-line interface, there are multiple tools and formats available for declaring the desired state of your clusters and applications - Helm, Kustomize, Carvel, Acorn, Timoni, Pulumi – each with pros and cons. <!-- You may have accessed the K8s API using REST calls or written applications using one of the K8s client libraries. Most K8s users have defined K8s objects as plain YAML or templated with Helm, Kustomize, or Carvel’s YTT. You may have also declared these using Infrastructure as Code (IaC), with languages like CUE, either directly or via tools like Acorn and Timoni. Alternatively, you can do this and other non-K8s cloud management using the language of your choice with Pulumi. --> Today I’ve decided to try Pulumi’s K8s integration to see how it stacks up. Pulumi has the distinct benefit of allowing me to create and manage K8s clusters and the applications that run on them using a single tool. ## What is Pulumi, though? 🏗️ Pulumi is an infrastructure as code (IaC) tool that lets you create, deploy, and manage cloud resources and apps using familiar programming languages like Python, JavaScript, Go, and more. Unlike traditional IaC tools primarily using declarative configuration files, Pulumi lets you define your infrastructure using code, providing the benefits of code reuse, version control, and programming language features. Pulumi supports managing K8s and other cloud resources on any cloud: AWS, Azure, Google Cloud, and many others. ## Ok, show me the code 👀 In this example, we'll use Pulumi to manage a K8s cluster on Google Cloud Platform (GCP) and deploy a public-facing K8s resource (a web page) within the cluster using Go. Let’s jump right in. ### Prerequisites 1. If you haven't already, install [Pulumi](https://www.pulumi.com/docs/install/), [Go](https://go.dev/doc/install), and [Google Cloud SDK](https://cloud.google.com/sdk/docs/downloads-interactive). 1. Create a GCP project, set up the free trial or billing info, and enable [Compute Engine API](https://console.cloud.google.com/apis/library/compute.googleapis.com) and [Artifact Registry API](https://console.cloud.google.com/marketplace/product/google/artifactregistry.googleapis.com). 3. Configure GCP Auth: ```sh $ gcloud auth login $ gcloud config set project <YOUR_GCP_PROJECT> $ gcloud auth application-default login ``` ⚠️ *Note that using Pulumi involves setting up authentication and configuration for your chosen cloud provider, which requires following that provider's documentation.* ### The app 1. Create a new directory for your app and navigate to it in your terminal: ```sh $ mkdir pulumi-demo-app && cd pulumi-demo-app ``` 1. Create `main.go`. For fun, this app prints a message from an environment variable if one exists, otherwise "hello world": ```go package main import ( "fmt" "os" ) func main() { message := os.Getenv("MESSAGE") if message == "" { message = "Hello, World!" } fmt.Println(message) } ``` 💡 You can test this with: ```console $ MESSAGE="Hello, Moon 🌘" go run main.go Hello, Moon 🌘 ``` 1. Create a `dockerfile` to run your app in a container: ```docker FROM golang:1.20 WORKDIR /app COPY . . RUN go build -o main . CMD ["./main"] ``` 💡 Test with: ```console $ docker build -t pulumi-demo-app . $ docker run -e MESSAGE="Hello, Saturn 🪐" pulumi-demo-app Hello, Saturn 🪐 ``` ⚠️ *Note if outside `$GOPATH`, add go mod so that `go build` in `dockerfile` will succeed:* ``` $ go mod init example.com/m $ go mod tidy ``` ### The cluster Now that we've checked our demo app works in a container, let's try it in K8s. First we'll need a cluster. <!-- https://www.pulumi.com/registry/packages/gcp/how-to-guides/gcp-go-gke/ --> <!-- https://github.com/pulumi/examples/blob/master/gcp-go-gke/main.go --> 1. In a new folder, create a new Go project with `pulumi new`: ```sh $ mkdir pulumi-demo-cluster && cd pulumi-demo-cluster $ pulumi new go Created project 'pulumi-demo-cluster' Created stack 'dev' Installing dependencies... Finished installing dependencies Your new project is ready to go! ✨ To perform an initial deployment, run `pulumi up` ``` 1. Paste pulumi code into `main.go`: ```go package main import ( "github.com/pulumi/pulumi-gcp/sdk/v5/go/gcp/container" "github.com/pulumi/pulumi/sdk/v3/go/pulumi" ) func main() { pulumi.Run(func(ctx *pulumi.Context) error { engineVersions, err := container.GetEngineVersions(ctx, &container.GetEngineVersionsArgs{}) if err != nil { return err } masterVersion := engineVersions.LatestMasterVersion _, err2 := container.NewCluster(ctx, "demo-cluster", &container.ClusterArgs{ InitialNodeCount: pulumi.Int(1), MinMasterVersion: pulumi.String(masterVersion), NodeVersion: pulumi.String(masterVersion), NodeConfig: &container.ClusterNodeConfigArgs{ // Only need a single node for dev demo MachineType: pulumi.String("n1-standard-1"), // Preemptible is fine for demo // See https://www.pulumi.com/registry/packages/gcp/api-docs/container/cluster/ Preemptible: pulumi.Bool(true), Preemptible: pulumi.Bool(true), OauthScopes: pulumi.StringArray{ pulumi.String("https://www.googleapis.com/auth/compute"), pulumi.String("https://www.googleapis.com/auth/devstorage.read_only"), pulumi.String("https://www.googleapis.com/auth/logging.write"), pulumi.String("https://www.googleapis.com/auth/monitoring"), }, }, }) if err2 != nil { return err2 } return nil }) } ``` 💡 *Note that for clarity this demo uses the minimum code required. See the <https://github.com/pulumi/examples/blob/master/gcp-go-gke/main.go> repo for a more detailed example.* 1. Update go.mod and download dependencies: ```go $ go mod tidy && go get ``` 1. Set the Pulumi configurations for your GCP project and zone: ```sh $ pulumi config set gcp:project <YOUR_GCP_PROJECT> $ pulumi config set gcp:zone <YOUR_GCP_ZONE> ``` 💡 *This will add configurations to a file called `Pulumi.dev.yaml`, where `dev` is a stack in your `pulumi-demo-cluster` project. Note you can set `gcp:zone` to any [valid GCP zone](https://cloud.google.com/compute/docs/regions-zones#available).* 1. Preview and deploy changes: ```console $ pulumi up Do you want to perform this update? yes ``` 💡 *Note `pulumi up` is idempotent, so run as many times as you wish. Selecting "no" aborts, and "details" gives a clear picture of what would happen next time you select "yes".* 💡 Test with: ```console $ kubectl get nodes NAME STATUS ROLES AGE VERSION pulumi-demo-cluster-***-default-pool-*** Ready <none> 10m v1.27.4-gke.900 ``` ### K8s resources Finally we'll need to define the K8s resources needed to run our app in the cluster. 1. Remember to push your container to Google [Artifact Registry](https://cloud.google.com/artifact-registry) so your Deployment can access it: ``` $ gcloud artifacts repositories create pulumi-demo-app \ --repository-format=docker \ --location=us-east1 \ --description="Container repository for Pulumi demo app" Created repository [pulumi-demo-app]. ``` 💡 *Test with `gcloud artifacts repositories list`.* ``` gcloud artifacts repositories list Listing items under project <YOUR_GCP_PROJECT>, across all locations. ARTIFACT_REGISTRY REPOSITORY FORMAT MODE DESCRIPTION LOCATION LABELS ENCRYPTION CREATE_TIME UPDATE_TIME SIZE (MB) pulumi-demo-app DOCKER STANDARD_REPOSITORY Container repository for Pulumi demo app us-east1 Google-managed key 2023-08-11T20:36:58 2023-08-11T20:36:58 0 ``` Configure Docker: ``` $ gcloud auth configure-docker \ us-east1-docker.pkg.dev ``` 💡 *This will configure gcloud as the credential helper for the Artifact Registry domain associated with this repository's location* Format your tag for Artifact Registry and push: ``` $ docker tag pulumi-demo-app \ us-east1-docker.pkg.dev/<YOUR_GCP_PROJECT>/pulumi-demo-app/pulumi-demo-app:v1 $ docker push us-east1-docker.pkg.dev/<YOUR_GCP_PROJECT>/pulumi-demo-app/pulumi-demo-app:v1 ``` 1. In order to define the resources with the Go SDK, we'll need to create a new k8sprovider inside the `pulummi.Run()` function in `main.go`. This also requires a generated kubeconfig. Here's the minimum code to create that: ```diff diff --git a/main.go b/main.go index d07eefc..8785e2e 100644 --- a/main.go +++ b/main.go @@ -2,6 +2,7 @@ package main import ( "github.com/pulumi/pulumi-gcp/sdk/v5/go/gcp/container" + "github.com/pulumi/pulumi-kubernetes/sdk/v3/go/kubernetes" "github.com/pulumi/pulumi/sdk/v3/go/pulumi" ) @@ -14,7 +15,7 @@ func main() { } masterVersion := engineVersions.LatestMasterVersion - _, err2 := container.NewCluster(ctx, "demo-cluster", &container.ClusterArgs{ + cluster, err := container.NewCluster(ctx, "demo-cluster", &container.ClusterArgs{ InitialNodeCount: pulumi.Int(1), MinMasterVersion: pulumi.String(masterVersion), NodeVersion: pulumi.String(masterVersion), @@ -32,6 +33,15 @@ func main() { }, }, }) + if err != nil { + return err + } + + ctx.Export("kubeconfig", generateKubeconfig(cluster.Endpoint, cluster.Name, cluster.MasterAuth)) + + _, err2 := kubernetes.NewProvider(ctx, "k8sprovider", &kubernetes.ProviderArgs{ + Kubeconfig: generateKubeconfig(cluster.Endpoint, cluster.Name, cluster.MasterAuth), + }, pulumi.DependsOn([]pulumi.Resource{cluster})) if err2 != nil { return err2 } @@ -39,3 +49,35 @@ func main() { return nil }) } + +func generateKubeconfig(clusterEndpoint pulumi.StringOutput, clusterName pulumi.StringOutput, + clusterMasterAuth container.ClusterMasterAuthOutput) pulumi.StringOutput { + context := pulumi.Sprintf("demo_%s", clusterName) + + return pulumi.Sprintf(`apiVersion: v1 +clusters: +- cluster: + certificate-authority-data: %s + server: https://%s + name: %s +contexts: +- context: + cluster: %s + user: %s + name: %s +current-context: %s +kind: Config +preferences: {} +users: +- name: %s + user: + exec: + apiVersion: client.authentication.k8s.io/v1beta1 + command: gke-gcloud-auth-plugin + installHint: Install gke-gcloud-auth-plugin for use with kubectl by following + https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke + provideClusterInfo: true +`, + clusterMasterAuth.ClusterCaCertificate().Elem(), + clusterEndpoint, context, context, context, context, context, context) +} ``` 3. Define a new Kubernetes Namespace in `pulumi.Run()`: ```diff diff --git a/main.go b/main.go index 8785e2e..8499d0d 100644 --- a/main.go +++ b/main.go @@ -3,6 +3,8 @@ package main import ( "github.com/pulumi/pulumi-gcp/sdk/v5/go/gcp/container" "github.com/pulumi/pulumi-kubernetes/sdk/v3/go/kubernetes" + corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v3/go/kubernetes/core/v1" + metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v3/go/kubernetes/meta/v1" "github.com/pulumi/pulumi/sdk/v3/go/pulumi" ) @@ -39,9 +41,18 @@ func main() { ctx.Export("kubeconfig", generateKubeconfig(cluster.Endpoint, cluster.Name, cluster.MasterAuth)) - _, err2 := kubernetes.NewProvider(ctx, "k8sprovider", &kubernetes.ProviderArgs{ + k8sProvider, err := kubernetes.NewProvider(ctx, "k8sprovider", &kubernetes.ProviderArgs{ Kubeconfig: generateKubeconfig(cluster.Endpoint, cluster.Name, cluster.MasterAuth), }, pulumi.DependsOn([]pulumi.Resource{cluster})) + if err != nil { + return err + } + + _, err2 := corev1.NewNamespace(ctx, "app-ns", &corev1.NamespaceArgs{ + Metadata: &metav1.ObjectMetaArgs{ + Name: pulumi.String("demo-ns"), + }, + }, pulumi.Provider(k8sProvider)) if err2 != nil { return err2 } ``` 1. Let's now define a Deployment for our app! ```diff diff --git a/main.go b/main.go index 8499d0d..5a8f091 100644 --- a/main.go +++ b/main.go @@ -3,6 +3,7 @@ package main import ( "github.com/pulumi/pulumi-gcp/sdk/v5/go/gcp/container" "github.com/pulumi/pulumi-kubernetes/sdk/v3/go/kubernetes" + appsv1 "github.com/pulumi/pulumi-kubernetes/sdk/v3/go/kubernetes/apps/v1" corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v3/go/kubernetes/core/v1" metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v3/go/kubernetes/meta/v1" "github.com/pulumi/pulumi/sdk/v3/go/pulumi" @@ -48,13 +49,53 @@ func main() { return err } - _, err2 := corev1.NewNamespace(ctx, "app-ns", &corev1.NamespaceArgs{ + namespace, err := corev1.NewNamespace(ctx, "app-ns", &corev1.NamespaceArgs{ Metadata: &metav1.ObjectMetaArgs{ Name: pulumi.String("demo-ns"), }, }, pulumi.Provider(k8sProvider)) - if err2 != nil { - return err2 + if err != nil { + return err + } + + appLabels := pulumi.StringMap{ + "app": pulumi.String("demo-app"), + } + _, err = appsv1.NewDeployment(ctx, "app-dep", &appsv1.DeploymentArgs{ + Metadata: &metav1.ObjectMetaArgs{ + Namespace: namespace.Metadata.Elem().Name(), + }, + Spec: appsv1.DeploymentSpecArgs{ + Selector: &metav1.LabelSelectorArgs{ + MatchLabels: appLabels, + }, + Replicas: pulumi.Int(3), + Template: &corev1.PodTemplateSpecArgs{ + Metadata: &metav1.ObjectMetaArgs{ + Labels: appLabels, + }, + Spec: &corev1.PodSpecArgs{ + Containers: corev1.ContainerArray{ + corev1.ContainerArgs{ + Name: pulumi.String("demo-app"), + Image: pulumi.String("us-east1-docker.pkg.dev/r6by-pocs/pulumi-demo-app/pulumi-demo-app:2.0.0"), + Env: corev1.EnvVarArray{ + corev1.EnvVarArgs{ + Name: pulumi.String("MESSAGE"), + Value: pulumi.String("Hello Mars 💘"), + }, + corev1.EnvVarArgs{ + Name: pulumi.String("PORT"), + Value: pulumi.String("9000"), + }, + }, + }}, + }, + }, + }, + }, pulumi.Provider(k8sProvider)) + if err != nil { + return err } return nil ``` 1. And finally we need a Service with Ingress: ```diff diff --git a/main.go b/main.go index 5a8f091..8cc30b9 100644 --- a/main.go +++ b/main.go @@ -98,6 +98,34 @@ func main() { return err } + service, err := corev1.NewService(ctx, "app-service", &corev1.ServiceArgs{ + Metadata: &metav1.ObjectMetaArgs{ + Namespace: namespace.Metadata.Elem().Name(), + Labels: appLabels, + }, + Spec: &corev1.ServiceSpecArgs{ + Ports: corev1.ServicePortArray{ + corev1.ServicePortArgs{ + Port: pulumi.Int(80), + TargetPort: pulumi.Int(9000), + }, + }, + Selector: appLabels, + Type: pulumi.String("LoadBalancer"), + }, + }, pulumi.Provider(k8sProvider)) + if err != nil { + return err + } + + ctx.Export("url", service.Status.ApplyT(func(status *corev1.ServiceStatus) *string { + ingress := status.LoadBalancer.Ingress[0] + if ingress.Hostname != nil { + return ingress.Hostname + } + return ingress.Ip + })) + return nil }) } ``` 1. Now wait a moment and open your app in your browser using the Service external IP, and you should see your new app! ## Cleanup 🧹 I'll feel remiss if I don't remind you to clean up your cloud resources. The principle of least surprise applies to your wallet as well as your apps! 1. Use the Google console, or `gcloud` CLI, to delete your demo cluster. 1. Log out locally with `gcloud auth revoke <YOUR_GCP_ACCOUNT>` or `--all` ![](https://hackmd.io/_uploads/r1EXkINhh.jpg) ## Summary 🤩 There we have it! You should now be able to use Pulumi create and manage a K8s cluster, as well as deploy and manage your applications as IaC. For more Pulumi related content by yours truly (coming soon!) follow me on GitHub: [@scottrigby](https://github.com/scottrigby)
Last changed by  
 
Scott Rigby
126

Read more

GCXX Blueprint

Terminologyglossary items from https://github.com/open-gitops/documents/blob/v1.0.0/GLOSSARY.mdContinuous Declarative Description Desired State Drift Reconciliation Software System State Store Feedback Principlesfrom https://github.com/open-gitops/documents/blob/v1.0.0/PRINCIPLES.md

3/3/2023
GitOps repo structure

I don’t know that this can or even should be standardized. I think repo organization Is subject to Conway’s law, especially with GitOps because your source repo is now an integral part of your overall system. - manage up - manage down However, I do think we can provide some good questions to ask, good rules of thumb, and common patterns that match common use cases for those just getting started or looking to iterate on their current approach. Proposed rules of thumb (axioms, not best practices per se): There should be no standard for GitOps repo structure GitOps tooling should be flexible enough to allow for iterating on repo structure

4/12/2022
GitOps Principles

Summary GitOps is a set of principles for operating and managing software systems. The aggregate of all configuration data for a system form its Desired State which is defined as data sufficient to recreate the system so that instances of the system are behaviourally indistinguishable. When using GitOps, the Desired State of a system or subsystem is defined declaratively as versioned, immutable data, and the running system's configuration is continuously derived from this data. GitOps principles were derived from modern software operations but are rooted in pre-existing and widely adopted best practices. These principles are: The principle of declarative desired state

5/5/2021
Read more from Scott Rigby
Published on HackMD