Using hosted Image Builder programmatically

OUTDATED, SEE https://docs.google.com/document/d/1JezrLQ4HVnMT8grhwuNbfQyiJaGbxF-yKwsWddyqTxU/edit?usp=sharing

https://source.redhat.com/groups/public/blog_bounty_program

Inspiration

• Building a customized Azure VM image with Image Builder and libvirt
• Announcing full support for new Red Hat Enterprise Linux image builder service
• Using the no-cost Developer Subscription with the new Red Hat Enterprise Linux Image Builder hosted service
• Build RHEL images for Azure with Image Builder
• Introducing the hosted beta experience Red Hat Enterprise Linux Image Builder
• Build a custom CentOS Stream 9 cloud image

The post starts here:

Image Builder is a new hosted Red Hat offering for building customized cloud images. In this blog post, I want to show you how to use it from the command line.

Introducing

Image Builder is a hosted service in Red Hat Hybrid Cloud Console allowing you to quickly assemble a customized image. Hybrid cloud solutions can be developed super-quickly using the service because it can simultaneously upload your image to all the most popular hyperscalers - AWS, Google Cloud and Azure. You can read more about it in our blog post announcing its general availability. In order to access it, you can use the no-cost developer subscription, we also have a blog post about that.

Why would I want to use Image Builder programmatically?

Although we made the graphical interface as easy to use as possible, if you are planning of making 10s, or even 1000s of images, the interface might become a hurdle. Fortunately, the service has an API that you can use to automate all your image building needs.

Authentication

Image Builder uses OAuth 2.0 for authorization. Firstly, you need to generate an offline token for Red Hat APIs on this page. This token cannot be used directly with Image Builder. Secondly, you need to exchange it for an access token. You can do that simply using curl, just remember to save your offline to a variable named OFFLINE_TOKEN prior running this command:

$ curl --silent \
    --request POST \
    --data grant_type=refresh_token \
    --data client_id=rhsm-api \
    --data refresh_token=$OFFLINE_TOKEN \
    https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token \
  | jq .

When you run this command, it should output something like this:

{
  "access_token": "oiZjo1Mjhk...",
  "expires_in": 900,
  "refresh_expires_in": 0,
  "refresh_token": "eyJhbG...",
  "token_type": "bearer",
  "not-before-policy": 0,
  "session_state": "f0dbb8d4-4e4e-4654-844c-6f3704c84422",
  "scope": "offline_access"
}

You can use jq to get the actual access token from the JSON payload and save it in a variable using the following snippet:

$ access_token=$( \
    curl --silent \
      --request POST \
      --data grant_type=refresh_token \
      --data client_id=rhsm-api \
      --data refresh_token=$OFFLINE_TOKEN \
      https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token \
    | jq -r .access_token \
  )

Note that the access token has an expiration time. If Image Builder returns an authorization error, just rerun the command to get a new one.

More details about Red Hat APIs and OAuth can be found in this article.

Getting the API documentation

In order to build images, you will use the Image Builder API. Hybrid Console handily embeds documentation for all of its APIs, Image Builder's one is available on this page.

This blog post is oriented on the CLI though, so let's also introduce a simple curl command to retrieve the API documentation from the command line:

$ curl --silent \
    --header "Authorization: Bearer $access_token" \
    https://console.redhat.com/api/image-builder/v1/openapi.json \
  | jq .

This command prints the API documentation in the form of OpenAPI 3 specification encoded in JSON.

Red Hat's code is open and so is Image Builder which means that you can also get the specification in a YAML format from our upstream repository.

Building a simple Guest Image

Now, we are ready to actually start building some images. Let's start with something basic: Building an up-to-date RHEL 9.0 Guest Image for x86_64 CPU architecture that can be used with libvirt or in OpenStack.

Firstly, we need to create a compose request:

$ cat >request.json <<END
{
  "image_name": "My up-to-date guest image",
  "distribution": "rhel-90",
  "image_requests": [
    {
      "architecture": "x86_64",
      "image_type": "guest-image",
      "upload_request": {
        "type": "aws.s3",
        "options": {}
      }
    }
  ]
}
END

The most important fields are the distribution, the CPU architecture and the image type. I set these to build a RHEL 9.0 x86_64 Guest image as promised. The upload request's type is set to aws.s3 as required by the API. The image can be also named to your liking, this example names the image "My up-to-date guest image".

Now that we have the request body prepared, we are ready to send a request to the Image Builder's API:

$ curl --silent \
    --request POST \
    --header "Authorization: Bearer $access_token" \
    --header "Content-Type: application/json" \
    --data @request.json \
    https://console.redhat.com/api/image-builder/v1/compose

If everything went alright, you should see output similar to this one:

{"id":"fd4ecf3c-f0ce-43dd-9fcc-6ad11208b939"}

Since we are going to need the ID to check the status of the image build, I suggest putting it directly into a variable with a small jq helper:

$ compose_id=$( \
    curl --silent \
      --request POST \
      --header "Authorization: Bearer $access_token" \
      --header "Content-Type: application/json" \
      --data @request.json \
      https://console.redhat.com/api/image-builder/v1/compose \
    | jq -r .id \
  )

If you now look in the Red Hat Hybrid Cloud Console, you can see that the image is indeed building:

Image builds take usually a few minutes, you can check the status using the following call:

$ curl \
    --silent \
    --header "Authorization: Bearer $access_token" \
    "https://console.redhat.com/api/image-builder/v1/composes/$compose_id" \
  | jq .

When you run this immediately after you created your compose, you should see the call output the following message:

{
  "image_status": {
    "status": "building"
  }
}

After some time, the returned message will change to:

{
  "image_status": {
    "status": "success",
    "upload_status": {
      "options": {
        "url": "https://image-builder-service-production.s3.amazonaws.com/composer-api-766eed27-bfa8-4801-806b-ad01e16e5a22-disk.qcow2?e42..."
      },
      "status": "success",
      "type": "aws.s3"
    }
  }
}

Yay, your image is ready. It's time to download it using the following command:

$ curl --location -O \
    https://image-builder-service-production.s3.amazonaws.com/composer-api-766eed27-bfa8-4801-806b-ad01e16e5a22-disk.qcow2?e42...

Now, you can run it locally using libvirt, or upload it to OpenStack. Note that the S3 link currently has an expiration time of 6 hours so don't be surprised if it's gone the next day.

Building a customized AWS image

The previous example was cool as it always gives you a fresh and up-to-date image. Let's try something more complex this time: Let's target AWS, add extra packages into the image, customize the partition layout and embed an activation key so you don't need to care about subscriptions.

You should already know everything about authentication and the OpenAPI specification so let's skip directly to the request:

$ cat >request.json <<EOF
{
  "image_name": "My customized AMI",
  "distribution": "rhel-86",
  "image_requests": [
    {
      "architecture": "x86_64",
      "image_type": "aws",
      "upload_request": {
        "type": "aws",
        "options": {
          "share_with_accounts": [
            "438669297788"
          ]
        }
      }
    }
  ],
  "customizations": {
    "packages": [
      "nginx"
    ],
    "filesystem": [
      {
        "mountpoint": "/var",
        "min_size": 10737418240
      }
    ],
    "subscription": {
      "organization": 123456789,
      "activation-key": "toucan",
      "server-url": "subscription.rhsm.redhat.com",
      "base-url": "http://cdn.redhat.com/",
      "insights": true
    }
  }
}
EOF

Let's go over the changes to the request:

Firstly, the distribution was switched to RHEL 8.6 to showcase that RHEL 8.x is also supported. In the image request, the image type was changed to aws which is also reflected in the upload request. Hosted Image Builder uploads the images to Red Hat's AWS account and then it shares the image with you, so the share_with_accounts field needs to be filled in order to tell Image Builder with which account to share the image. Note that these images will go away after 14 days so if you want to use it for longer, you have to make a copy.

In a comparison with the previous request, a big customizations object was added:

• nginx will be preinstalled in the image
• The root partition will be 4 GiB big. /var will be on a separate, 10GiB big partition.
• Instances spun up from the image will be automatically subscribed using the given activation key. If you don't know how to create one, there's an article about the process.

Let's send the request:

$ compose_id=$( \
    curl --silent \
      --request POST \
      --header "Authorization: Bearer $access_token" \
      --header "Content-Type: application/json" \
      --data @request.json \
      https://console.redhat.com/api/image-builder/v1/compose \
    | jq -r .id \
  )

and start checking its status:

$ curl --silent \
    --header "Authorization: Bearer $access_token" \
    "https://console.redhat.com/api/image-builder/v1/composes/$compose_id" \
  | jq .

Once the image build is finished (it can take a bit longer as import to AWS tends to take few minutes), you should get the following response:

{
  "image_status": {
    "status": "success",
    "upload_status": {
      "options": {
        "ami": "ami-01f2c869485288e5e",
        "region": "us-east-1"
      },
      "status": "success",
      "type": "aws"
    }
  }
}

If you open the same image in the Red Hat Hybrid Cloud Console, you should see something like this:

You can launch your new image using the link in the Console. This article is about the command line though, so let's use AWS CLI to launch an instance and find its public IP address:

$ aws --region us-east-1 \
    ec2 run-instances \
    --image-id ami-01f2c869485288e5e \
    --key-name KEY_NAME
$ aws --region us-east-1 \
    ec2 describe-instances \
    --instance-ids INSTANCE_ID \
    --query 'Reservations[*].Instances[*].PublicIpAddress' \
    --output text
3.83.39.87

When you ssh into the instance using ssh ec2-user@3.83.39.87, you can confirm that the instance has the requested disk layout, nginx is installed and new packages can be immediately installed because the machine is subscribed:

$ lsblk
NAME              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda              202:0    0   15G  0 disk 
|-xvda1           202:1    0    1M  0 part 
|-xvda2           202:2    0   14G  0 part 
| |-rootvg-rootlv 253:0    0    4G  0 lvm  /
| `-rootvg-varlv  253:1    0   10G  0 lvm  /var
`-xvda3           202:3    0  512M  0 part /boot
xvdc              202:32   0  896M  0 disk [SWAP]
$ rpm -q nginx
nginx-1.14.1-9.module+el8.0.0+4108+af250afe.x86_64
$ sudo dnf install tmux
Updating Subscription Management repositories.
Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)                                  6.7 MB/s |  47 MB     00:07    
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)                                10 MB/s |  44 MB     00:04    
Last metadata expiration check: 0:00:03 ago on Fri May 27 12:25:55 2022.
Dependencies resolved.
=======================================================================================================================
 Package             Architecture          Version                  Repository                                    Size
=======================================================================================================================
Installing:
 tmux                x86_64                2.7-1.el8                rhel-8-for-x86_64-baseos-rpms                317 k

Transaction Summary
=======================================================================================================================
Install  1 Package

Total download size: 317 k
Installed size: 770 k
Is this ok [y/N]:

Note that this section about AWS CLI might not work for you if your default VPC, default subnet or default security group is set differently.

Conclusion

Hosted Image Builder can very simply be used via its API. You just need a few curl commands and you quickly get up-to-date and customizable RHEL images directly into a cloud of your choice. Start using it today at https://console.redhat.com/insights/image-builder/.