Google Cloud "GCP" native NixOS images build

--- tags: Publication --- # Google Cloud "GCP" native NixOS images build I am using NixOs as a Development environment that is running on Google Cloud VM. [NixOs official Web site](https://nixos.org/download.html) does not provide an official GCP image. On the NixOs Wiki page you can find an article how you can build your [GCP NixOs Image](https://nixos.wiki/wiki/Install_NixOS_on_GCE) that didn't work well in the past but now seems all build image issue was resolved and now works well. NixOS Wiki page article also provides two GCP storage that you can find outdated GCP VM images that seem not actively maintained anymore. _nixos-images > gs://nixos-images_ _nixos-cloud-images > gs://nixos-images_ In the below HowTo, we will cover how you can build your images in GCP by using Google Cloud VM and store the images in your Google storage bucket. ## Prepare Google Cloud environment. Before we can start, you will need to have a [Google Cloud Account](https://console.cloud.google.com/) We will create a dedicated GCP Project with VM that we will use to build updated NixOs images from the latest nixpkgs builds. ### Prerequisites Before starting In the 1st step, you will need to install [Google Cloud SDK](https://cloud.google.com/sdk) Depending on the OS or Linux Distribution, and please follow installation instructions. ### Gcloud authentication Once installation is complete, you will need to authenticate CLI to be able to access your google cloud resources. ```shell= ~> gcloud auth login Go to the following link in your browser: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=32542940657.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&state=skHTaEGrhJSYIzDIwt4phrhSzUm97t&prompt=consent&access_type=offline&code_challenge=EcgLC0aZhpefFkL7k6ep-8lvJ1Og8NgCvs9VllOE5lQ&code_challenge_method=S256 Enter verification code: ``` ## GCP Project creation Once authenticated, you should be able to access all cloud resources in your google cloud account. If you are using your account for the 1st time, Google will create a default Project for your account. We will ignore the default project and will make a dedicated project to build NioxOs images. ### GCP new Project creation We will create a dedicated GCP project. I use the name for the project mudrii-nixos ```shell= ~> gcloud projects create mudrii-nixos Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/mudrii-nixos]. Waiting for [operations/cp.8533728690478532386] to finish...done. Enabling service [cloudapis.googleapis.com] on project [mudrii-nixos]... Operation "operations/acf.p2-237810657129-fa1f0fa7-3015-46ff-ab52-f92d7bd3f8df" finished successfully. ``` Get details on the created project. ```shell= ~> gcloud projects describe mudrii-nixos createTime: '2021-09-06T08:57:16.001Z' lifecycleState: ACTIVE name: mudrii-nixos projectId: mudrii-nixos projectNumber: '111111111111' ``` ```shell= ~> gcloud projects list PROJECT_ID NAME PROJECT_NUMBER mudrii-nixos mudrii-nixos 111111111111 ``` ### Configure gcloud CLI Once a new project was created is a good practice to add the newly created project as default for gcloud CLI. every command executed with gcloud of gsutils will perform on the newly created project ```shell= ~> gcloud config set project mudrii-nixos Updated property [core/project]. ``` ### Configure Project billing One more step is needed to make the GCP project usable we will need to add billing to the project. #### List available billing accounts ```shell= ~> gcloud alpha billing accounts list ACCOUNT_ID NAME OPEN MASTER_ACCOUNT_ID ZZZZZ-ZZZZZZZ-ZZZZZZ MyBill True ``` #### Associate billing account with the project ```shell= ~> gcloud alpha billing accounts projects link mudrii-nixos --billing-account=ZZZZZ-ZZZZZZZ-ZZZZZZ billingAccountName: billingAccounts/ZZZZZ-ZZZZZZZ-ZZZZZZ billingEnabled: true name: projects/mudrii-nixos/billingInfo projectId: mudrii-nixos ``` #### Verify billing account association ```shell= ~> gcloud alpha billing accounts projects list --billing-account=ZZZZZ-ZZZZZZZ-ZZZZZZ PROJECT_ID BILLING_ACCOUNT_ID BILLING_ENABLED mudrii-nixos ZZZZZ-ZZZZZZZ-ZZZZZZ True ``` ## Create and configure Google storage bucket The next step is to create a Google storage bucket to store generated nixos GCP VM images. ```shell= ~> gsutil mb gs://nixos-images-gcp gsutil mb gs://nixos-images-gcp ~> gsutil du -s -h gs://nixos-images-gcp ``` I decided to grant read access to external users to the generated images. If you do not intend to share generated images, you can skip the below step. ```shell= ~> gsutil iam ch allUsers:objectViewer gs://nixos-images-gcp ``` ## Create and Configure GCP networking By default, Google will create a global VPC and subnets for every region and firewall rules once a project is created. To make it clean, I decided to create a separate network stack with a subnet and firewall to control the resources better. Before creating network stuck a good practice to remove all automated default network VPC, subnets, firewall. ### List available network resource #### List VPC ```shell= ~> gcloud compute networks list NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4 default AUTO REGIONAL ``` #### List subnets ```shell= ~> gcloud compute networks subnets list NAME REGION NETWORK RANGE default us-central1 default 10.128.0.0/20 default europe-west1 default 10.132.0.0/20 default us-west1 default 10.138.0.0/20 default asia-east1 default 10.140.0.0/20 default us-east1 default 10.142.0.0/20 default asia-northeast1 default 10.146.0.0/20 default asia-southeast1 default 10.148.0.0/20 default us-east4 default 10.150.0.0/20 default australia-southeast1 default 10.152.0.0/20 default europe-west2 default 10.154.0.0/20 default europe-west3 default 10.156.0.0/20 default southamerica-east1 default 10.158.0.0/20 default asia-south1 default 10.160.0.0/20 default northamerica-northeast1 default 10.162.0.0/20 default europe-west4 default 10.164.0.0/20 default europe-north1 default 10.166.0.0/20 default us-west2 default 10.168.0.0/20 default asia-east2 default 10.170.0.0/20 default europe-west6 default 10.172.0.0/20 default asia-northeast2 default 10.174.0.0/20 default asia-northeast3 default 10.178.0.0/20 default us-west3 default 10.180.0.0/20 default us-west4 default 10.182.0.0/20 default asia-southeast2 default 10.184.0.0/20 default europe-central2 default 10.186.0.0/20 default northamerica-northeast2 default 10.188.0.0/20 default asia-south2 default 10.190.0.0/20 default australia-southeast2 default 10.192.0.0/20 ``` #### List firewalls ```shell= ~> gcloud compute firewall-rules list NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED default-allow-icmp default INGRESS 65534 icmp False default-allow-internal default INGRESS 65534 tcp:0-65535,udp:0-65535,icmp False default-allow-rdp default INGRESS 65534 tcp:3389 False default-allow-ssh default INGRESS 65534 tcp:22 False To show all fields of the firewall, please show in JSON format: --format=json To show all fields in table format, please see the examples in --help. ``` ### Remove default created network resource Before removing VPC, we need to remove the firewall 1st, and once all firewall rules have been removed, we can remove the default VPC. #### Remove default Firewall ```shell= ~> gcloud compute firewall-rules delete default-allow-internal --quiet Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-internal]. ~> gcloud compute firewall-rules delete default-allow-icmp --quiet Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-icmp]. ~> gcloud compute firewall-rules delete default-allow-rdp --quiet Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-rdp]. ~> gcloud compute firewall-rules delete default-allow-ssh --quiet Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/default-allow-ssh]. ``` #### Remove default VPC ```shell= ~> gcloud compute networks delete default --quiet Deleted [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/networks/default]. ``` ### Create Networking resources We cleared all default network stuck, and now we can create a new network stuck. #### Create VPC Now we are ready to create a custom VPC. ```shell= ~> gcloud compute networks create vpc-nixos --subnet-mode=custom Created [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/networks/vpc-nixos]. NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4 vpc-nixos CUSTOM REGIONAL Instances on this network will not be reachable until firewall rules are created. As an example, you can allow all internal traffic between instances as well as SSH, RDP, and ICMP by running: ``` #### Create custom subnet In the below example, I use a narrow IP range in asia-southeast1 “Singapore” region. As I will run a single instance that will generate nixos images, I do not intend to run anything else. ```shell= ~> gcloud compute networks subnets create subnet-sg --network=vpc-nixos --range=192.168.1.0/24 --region asia-southeast1 Created [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/regions/asia-southeast1/subnetworks/subnet-sg]. NAME REGION NETWORK RANGE subnet-sg asia-southeast1 vpc-nixos 192.168.1.0/24 ``` #### Create a firewall rule to allow ssh In order to connect to remote VM over ssh we need to open a firewall on port 22. ```shell= ~> gcloud compute --project=mudrii-nixos firewall-rules create allow-ssh --direction=INGRESS --priority=1000 --network=vpc-nixos --action=ALLOW --rules=tcp:22 --target-tags=allow-ssh Creating firewall...⠹Created [https://www.googleapis.com/compute/v1/projects/mudrii-nixos/global/firewalls/allow-ssh]. Creating firewall...done. NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED allow-ssh vpc-nixos INGRESS 1000 tcp:22 False ``` ## Add image to your VM images repository to be able to create NixOs VM in GCP To create VM with NixOS installed, we need to have a VM image in our GCP images repositories. In the below example, I am using one of the images I created previously locate in `gs://nixos-images-gcp/` _Note: Make sure to specify your project ID `--project=mudrii-nixos`_ ```shell= ~> gcloud compute images create nixos-21-05-2873 \ --source-uri=gs://nixos-images-gcp/nixos-image-21.05.2873.6bfe71f2a4e-x86_64-linux.raw.tar.gz \ --description=nixos-image-21.05.2873.6bfe71f2a4e \ --family=nixos \ --project=mudrii-nixos ``` List available images in your image repository ```shell= ~> gcloud compute images list | grep nixos nixos-image-21-05-2873-6bfe71f2a4e-x86-64-linux mudrii-nixos nixos-image-21-05 READY ``` ## Create NixOS VM Our network stuck is up a ready we can create a dedicated VM to build nixos GCP images. Few point to mention. * Vm will be created in the same region where we set our subnet. `--zone=asia-southeast1-b` * `--machine-type=n2d-standard-4` VM AMD EMYC 4 vCPU and 16 GB RAM * `--metadata=enable-oslogin=true` allow you to ssh into the VM with your gcloud account * `--tags=allow-ssh` Added tag to the VM to allow open port ssh we configured in the firewall * `--boot-disk-size=60GB` size of the Disk you may select a lower capacity I find 30 works well too * `--boot-disk-type=pd-ssd` This is the fastest option local SSD will speed up significantly image generation ```shell= ~> gcloud beta compute --project=mudrii-nixos instances create nixos-base --zone=asia-southeast1-b --machine-type=n2d-standard-4 --subnet=subnet-sg --network-tier=PREMIUM --metadata=enable-oslogin=true --tags=allow-ssh --image=nixos-21-05-2873 --image-project=mudrii-nixos --boot-disk-size=60GB --boot-disk-type=pd-ssd --boot-disk-device-name=nixos-base Created [https://www.googleapis.com/compute/beta/projects/mudrii-nixos/zones/asia-southeast1-b/instances/nixos-base]. WARNING: Some requests generated warnings: - Disk size: '60 GB' is larger than image size: '3 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details. NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS nixos-base asia-southeast1-b n2d-standard-4 192.168.1.2 35.188.151.168 RUNNING ``` ## SSH into remote We can ssh directly to the newly created VM with the `gcloud` command ```shell= ~> gcloud compute ssh --project=mudrii-nixos nixos-base ``` ## Building NixOS image Once SSH gets into the newly created NixOs Google VM, we can start preparing to build the NixoS CGP image. ### Authenticate from VM into Google CLoud First, we need to become root. ```shell= ~> sudo -i ``` We need access to google SDK to upload newly created images into the google bucket created in the previous steps. To authenticate, we need google SDK, and instead of installing, we will use `nix-shell`. ```shell= [root@nixos-base:~]# nix-shell -p google-cloud-sdk git ``` Once we have access to gcloud binary we can auth into the google cloud ```shell= [nix-shell:~]# gcloud auth login [nix-shell:~]# gcloud projects list PROJECT_ID NAME PROJECT_NUMBER mudrii-nixos mudrii-nixos 111111111111 [nix-shell:~]# gcloud config set project mudrii-nixos Updated property [core/project]. [nix-shell:~]# gcloud config configurations list NAME IS_ACTIVE ACCOUNT PROJECT COMPUTE_DEFAULT_ZONE COMPUTE_DEFAULT_REGION default True my_email@gmail.com mudrii-nixos ``` To validate authentication, you can check if you have access to your Gcloud bucket created in the previous step. _Note: Make sure you add `BOTO_CONFIG=/dev/null` or you will get an error during the image build_ ```shell= export BOTO_CONFIG=/dev/null [nix-shell:~]# gsutil ls -l gs://nixos-images-gcp ``` ### Update NixOS to the latest version To create an image with the latest stable NixOs is essential to update the existing version with the latest version. Check existing NixOs version and Linux Kernel. ```shell= [nix-shell:~]# nixos-version 21.05.3001.12eb1d16ae3 (Okapi) [nix-shell:~]# uname -a Linux nixos-base.asia-southeast1-b.c.mudrii-nixos.internal 5.10.62 #1-NixOS SMP Fri Sep 3 08:09:31 UTC 2021 x86_64 GNU/Linux ``` #### Update NixOs to the latest version ##### Update 1st nix-channels ```shell= [nix-shell:~]# nix-channel --list nixos https://nixos.org/channels/nixos-21.05 [nix-shell:~]# nix-channel --update unpacking channels... created 1 symlinks in user environment ``` ##### Update 1st NixOS ```shell= [nix-shell:~]# nixos-rebuild switch building Nix... building the system configuration... these derivations will be built: ... .. . [nix-shell:~]# nixos-version 21.05.3021.8b0b81dab17 (Okapi) ``` Or if the version is outdated significantly, run ```sh [nix-shell:~]# nixos-rebuild switch --upgrade ``` ### NixOs cleanup Before attempting image creation is a good idea to do a cleanup and do a garbage collection on existing update and repair any sha inconsistency that may cause failed image build. ```shell= [nix-shell:~]# nix-collect-garbage -d removing old generations of profile /nix/var/nix/profiles/system removing generation 4 removing old generations of profile /nix/var/nix/profiles/per-user/root/channels removing generation 5 finding garbage collector roots... removing stale link from ... .. . deleting '/nix/store/trash' deleting unused links... note: currently hard linking saves 21.60 MiB 167 store paths deleted, 1129.80 MiB freed [nix-shell:~]# nix-store --gc finding garbage collector roots... deleting garbage... deleting '/nix/store/trash' deleting unused links... note: currently hard linking saves 21.60 MiB 0 store paths deleted, 0.00 MiB freed [nix-shell:~]# nix-store --repair --verify --check-contents reading the Nix store... checking path existence... checking hashes... path '/nix/store/kacsvbh8qjl28izy5g7a8p96z6xdvnik-google-cloud-sdk-340.0.0' was modified! expected hash 'sha256:0dvxzzklaswx0d2svx0nzjilqfmgd2dxffi7hcbz89p7r6w1jab0', got 'sha256:1wpsb25jajbqvsw29jy073w0cr994005xb22pb9p8mkinn9vx8hp' copying path '/nix/store/kacsvbh8qjl28izy5g7a8p96z6xdvnik-google-cloud-sdk-340.0.0' from 'https://cache.nixos.org'... [nix-shell:~]# nix-store --optimise 430.37 MiB freed by hard-linking 47107 files ``` ### NixOs GCP image build To build a GCP nixos image, we need to clone the nixpkgs repository where build scripts is locate ```shell= [nix-shell:~]# git clone https://github.com/NixOS/nixpkgs.git --depth 1 Cloning into 'nixpkgs'... remote: Enumerating objects: 45763, done. remote: Counting objects: 100% (45763/45763), done. remote: Compressing objects: 100% (29848/29848), done. remote: Total 45763 (delta 1593), reused 38844 (delta 1293), pack-reused 0 Receiving objects: 100% (45763/45763), 30.46 MiB | 14.19 MiB/s, done. Resolving deltas: 100% (1593/1593), done. Updating files: 100% (27800/27800), done. ``` The final step is to run a script that will build a GCP nixos image based on the underlying system version. _Note: Make sure you specify your GCP storage bucket name in `BUCKET_NAME=` ```shell= [nix-shell:~]# BUCKET_NAME=nixos-images-gcp nixpkgs/nixos/maintainers/scripts/gce/create-gce.sh these paths will be fetched (0.05 MiB download, 0.28 MiB unpacked): /nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev copying path '/nix/store/p5lnl4zr45n7mf9kz9w8yz3rqh001b5c-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'... ... .. . /nix/store/ii2h0jqwfzmzdc6lxyfmg4ia5726r6g6-google-compute-image gs://nixos-images-gcp/nixos-image-21.05.3021.8b0b81dab17-x86_64-linux.raw.tar.gz ``` ### NixOs image build verification Once the image is created and uploaded into Google storage bucket and in the image repository, we can verify by; ```shell= [nix-shell:~]# gsutil ls -l gs://nixos-images-gcp 434150003 2021-09-12T03:18:26Z gs://nixos-images-gcp/nixos-image-21.05.3021.8b0b81dab17-x86_64-linux.raw.tar.gz [nix-shell:~]# ``` Verify Image repository ```shell= [nix-shell:~]# gcloud compute images list | grep nixos nixos-image-21-05-3021-8b0b81dab17-x86-64-linux mudrii-nixos nixos-image-21-05 READY ``` ### Post-build cleanup It is good to remove the nixpkgs repository to minimize space and speed on the next build and stop VM to reduce the cost on your cloud bill. ```shell= [nix-shell:~]# ls -la total 44 drwx------ 8 root root 4096 Sep 13 11:02 . drwxr-xr-x 17 root root 4096 Sep 6 10:55 .. -rw------- 1 root root 2538 Sep 10 10:27 .bash_history drwxr-xr-x 3 root root 4096 Sep 6 11:07 .cache drwxr-xr-x 3 root root 4096 Sep 6 11:07 .config lrwxrwxrwx 1 root root 64 Sep 13 11:02 gce -> /nix/store/ii2h0jqwfzmzdc6lxyfmg4ia5726r6g6-google-compute-image drwxr-xr-x 3 root root 4096 Sep 6 11:48 .gsutil -rw-r--r-- 1 root root 45 Aug 25 05:05 .nix-channels drwx------ 2 root root 4096 Sep 13 10:30 .nix-defexpr drwxr-xr-x 9 root root 4096 Sep 13 10:55 nixpkgs drwx------ 2 root root 4096 Sep 6 10:55 .ssh [nix-shell:~]# rm -rf ~/nixpkgs [nix-shell:~]# sudo shutdown now Connection to 35.198.250.170 closed by remote host. Connection to 35.198.250.170 closed. ERROR: (gcloud.compute.ssh) [/run/current-system/sw/bin/ssh] exited with return code [255]. ``` ## Fin Next time when you want to create a new VM on Google Cloud you can specify in custom images your latest nixos image. Make sure you add `--metadata=enable-oslogin=true` allow you to ssh into the VM. ## Automated script ```shell= #! /usr/bin/env nix-shell #! nix-shell -i bash -p google-cloud-sdk git gcloud config configurations list export BOTO_CONFIG=/dev/null gsutil ls -l gs://nixos-images-gcp nixos-version uname -a nix-channel --list nix-channel --update nixos-rebuild switch nixos-rebuild switch --upgrade nixos-version nix-collect-garbage -d nix-store --gc nix-store --repair --verify --check-contents nix-store --optimise git clone https://github.com/NixOS/nixpkgs.git --depth 1 BUCKET_NAME=nixos-images-gcp nixpkgs/nixos/maintainers/scripts/gce/create-gce.sh gsutil ls -l gs://nixos-images-gcp gcloud compute images list | grep nixos ls -la rm -rf ~/nixpkgs ls -la exit ```

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.