Day 1 - part B: Building an image

tags: semiotmic2022

based on
https://training.play-with-docker.com/ops-s1-images/

The code of this section is here

If you are running Docker online: https://labs.play-with-docker.com/ you can upload files in the session terminal by dragging over it.

Basic steps

First thing you may want to do is figure out how to create our own images. While there are over 8 millions images (as of January 2022) on Docker Hub, it is almost certain that none of them are exactly what you run in your data center today. Even something as common as a Windows OS image would get its own tweaks before you actually run it in production.

We will start with the simplest form of image creation, in which we simply commit one of our container instances as an image. Then we will explore a much more powerful and useful method for creating images, the Dockerfile.

---

An important distinction with regard to images is between base images and child images.

• Base images are images that have no parent images, usually images with an OS like ubuntu, alpine or debian.
• Child images are images that are built on base images and add additional functionality.

Another key concept is the idea of official images and user images. (Both of which can be base images or child images.)

• Official images are Docker authorized images. There is dedicated team that is responsible for reviewing and publishing all Official Repositories content. This team works in collaboration with upstream software maintainers, security experts, and the broader Docker community. These are not prefixed by an organization or user name. Images like python, node, alpine and nginx are official (base) images.

To find out more about them, check out the Official Images Documentation.

• User images are images created and shared by users like you. They build on base images and add additional functionality. Typically these are formatted as user/image-name. The user value in the image name is your Docker Store user or organization name.

---

Image creation from a container

Let’s start by running an interactive shell in a ubuntu container:

$ docker run -it ubuntu bash

As you know from before, you just grabbed the image called “ubuntu” from Docker Store and are now running the bash shell inside that container.

To customize things a little bit we will install a package called figlet in this container. Your container should still be running so type the following commands at your ubuntu container command line:

apt-get update
apt-get install -y figlet
figlet "hello docker"

You should see the words “hello docker” printed out in large ascii characters on the screen. Go ahead and exit from this container

exit

Now let us pretend this new figlet application is quite useful and you want to share it with the rest of your team. You could tell them to do exactly what you did above and install figlet in to their own container, which is simple enough in this example. But if this was a real world application where you had just installed several packages and run through a number of configuration steps the process could get cumbersome and become quite error prone. Instead, it would be easier to create an image you can share with your team.

To start, we need to get the ID of this container using the ls command (do not forget the -a option as the non running container are not returned by the ls command).

$ docker container ls -a

Before we create our own image, we might want to inspect all the changes we made. Try typing the command

$ docker container diff <container ID>

for the container you just created. You should see a list of all the files that were added (A) to or changed (C ) in the container when you installed figlet. Docker keeps track of all of this information for us; this is part of the layer concept.

Now, to create an image we need to “commit” this container. Commit creates an image locally on the system running the Docker engine. Run the following command, using the container ID you retrieved, in order to commit the container and create an image out of it.

$ docker container commit CONTAINER_ID

That’s it - you have created your first image! Once it has been commited, we can see the newly created image in the list of available images.

$ docker image ls

You should see something like this:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
<none>              <none>              a104f9ae9c37        46 seconds ago      160MB
ubuntu              latest              14f60031763d        4 days ago          120MB

Note that the image we pulled down in the first step (ubuntu) is listed here along with our own custom image. Except our custom image has no information in the REPOSITORY or TAG columns, which would make it tough to identify exactly what was in this container if we wanted to share amongst multiple team members.

Adding this information to an image is known as tagging an image. From the previous command, get the ID of the newly created image and tag it so it’s named ourfiglet:

$ docker image tag <IMAGE_ID> ourfiglet
$ docker image ls

Now we have the more friendly name “ourfiglet” that we can use to identify our image.

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ourfiglet           latest              a104f9ae9c37        5 minutes ago       160MB
ubuntu              latest              14f60031763d        4 days ago          120MB

Here is a graphical view of what we just completed:

Now we will run a container based on the newly created ourfiglet image:

$ docker run ourfiglet figlet hello

As the figlet package is present in our ourfiglet image, the command returns the following output:

 _          _ _
| |__   ___| | | ___
| '_ \ / _ \ | |/ _ \
| | | |  __/ | | (_) |
|_| |_|\___|_|_|\___/

This example shows that we can create a container, add all the libraries and binaries in it and then commit it in order to create an image. We can then use that image just as we would for images pulled down from the Docker Store. We still have a slight issue in that our image is only stored locally. To share the image we would want to push the image to a registry somewhere. We'll see how to do this later…

This approach of manually installing software in a container and then committing it to a custom image is just one way to create an image. It works fine and is quite common. However, there is a more powerful way to create images. In the following exercise we will see how images are created using a Dockerfile, which is a text file that contains all the instructions to build an image.

Image creation using a Dockerfile

Instead of creating a static binary image, we can use a file called a Dockerfile to create an image. The final result is essentially the same, but with a Dockerfile we are supplying the instructions for building the image, rather than just the raw binary files. This is useful because it becomes much easier to manage changes, especially as your images get bigger and more complex.

Dockerfiles are powerful because they allow us to manage how an image is built, rather than just managing binaries. In practice, Dockerfiles can be managed the same way you might manage source code: they are simply text files so almost any version control system can be used to manage Dockerfiles over time.

We will use a simple example in this section and build a “hello world” application in Node.js. Do not be concerned if you are not familiar with Node.js; Docker (and this exercise) does not require you to know all these details.

We will start by creating a file in which we retrieve the host name and display it.

Type the following content into a file named index.js:

var os = require("os");
var hostname = os.hostname();
console.log("hello from " + hostname);

The file we just created is the javascript code for our server. As you can probably guess, Node.js will simply print out a “hello” message. We will:

• Docker-ize this application by creating a Dockerfile; we will use alpine as the base OS image,
• add a Node.js runtime and then
• copy our source code in to the container.
• We will also specify the default command to be run upon container creation.

Create a file named Dockerfile and copy the following content into it:

FROM alpine
RUN apk update && apk add nodejs
COPY . /app
WORKDIR /app
CMD ["node","index.js"]

Let’s build our first image out of this Dockerfile and name it hello:v0.1:

$ docker image build -t hello:v0.1 .

This is what you just completed:

We then start a container to check that our applications runs correctly:

$ docker run hello:v0.1

You should then have an output similar to the following one (the ID will be different though).

hello from 92d79b6de29f

What just happened? We created two files: our application code (index.js) is a simple bit of javascript code that prints out a message. And the Dockerfile is the instructions for Docker engine to create our custom container. This Dockerfile does the following:

1. Specifies a base image to pull FROM - the alpine image we used in earlier labs.
2. Then it RUNs two commands (apk update and apk add) inside that container which installs the Node.js server.
3. Then we told it to COPY files from our working directory in to the container. The only file we have right now is our index.js.
4. Next we specify the WORKDIR - the directory the container should use when it starts up
5. And finally, we gave our container a command (CMD) to run when the container starts.

Recall that in previous labs we put commands like echo "hello world" on the command line. With a Dockerfile we can specify precise commands to run for everyone who uses this container. Other users do not have to build the container themselves once you push your container up to a repository (which we will cover later) or even know what commands are used. The Dockerfile allows us to specify how to build a container so that we can repeat those steps precisely everytime and we can specify what the container should do when it runs. There are actually multiple methods for specifying the commands and accepting parameters a container will use, but for now it is enough to know that you have the tools to create some pretty powerful containers.

An example with a REST server

This example will build a basic REST server based on Flask. Flask is a “micro” web application framework written in Python. It allows writing simple and lightweight REST APIs.

The server we will design will handle a set of sensors, their creation, visualization, and update using an SQLite database.

We'll do this by first pulling together the components for a REST based server built with Python Flask, then dockerizing it by writing a Dockerfile. Finally, we'll build the image, and then run it.

The Python Flask app

We have to create the following files:

• the main REST API in file api.py
• an example of a sensor data updater in file add.py
• the Dockerfile

File api.py

#!/usr/bin/python
import sqlite3
from flask import Flask, request, jsonify

def connect_to_db():
    conn = sqlite3.connect('database.db')
    return conn

def insert_sensor(sensor):
    inserted_sensor = {}
    try:
        conn = connect_to_db()
        conn.row_factory = sqlite3.Row
        cur = conn.cursor()
        cur.execute("INSERT INTO sensors (tstamp, label, value, unity) VALUES (?, ?, ?, ?)", 
                    (sensor['tstamp'], sensor['label'], sensor['value'], sensor['unity']) )
        conn.commit()
        inserted_sensor = get_sensor_by_id(cur.lastrowid)
    except:
        conn().rollback()
    finally:
        conn.close()
    return inserted_sensor

def get_sensors():
    sensors = []
    try:
        conn = connect_to_db()
        conn.row_factory = sqlite3.Row
        cur = conn.cursor()
        cur.execute("SELECT * FROM sensors")
        rows = cur.fetchall()

        # convert row objects to dictionary
        for i in rows:
            sensor = {}
            sensor["label"]  = i["label"]
            sensor["value"]  = i["value"]
            sensor["unity"]  = i["unity"]
            sensor["tstamp"] = i["tstamp"]
            sensors.append(sensor)
    except:
        sensors = []
    return sensors

def get_sensor_by_id(sensor_id):
    sensors = []
    try:
        conn = connect_to_db()
        conn.row_factory = sqlite3.Row
        cur = conn.cursor()
        cur.execute("SELECT * FROM sensors WHERE label = ?", (sensor_id,))
        rows = cur.fetchall()

        # convert row objects to dictionary
        for i in rows:
            sensor = {}
            sensor["label"]  = i["label"]
            sensor["value"]  = i["value"]
            sensor["unity"]  = i["unity"]
            sensor["tstamp"] = i["tstamp"]
            sensors.append(sensor)
    except:
        sensors = []
    return sensors

def create_db_table():
    initsensors = [
    {
        "tstamp": "2009-06-15T13:45:30",
        "label": "arduino_nano33",
        "value": 25,
        "unity": "Celsius degrees"
    },
    {
        "tstamp": "2010-06-15T13:45:30",
        "label": "portenta_h7",
        "value": 0.123,
        "unity": "mWatts"
    },
    {
        "tstamp": "2011-06-15T13:45:30",
        "label": "raspberry4",
        "value": 4,
        "unity": "roentgen"
    }
    ]

    try:
        conn = connect_to_db()
        conn.execute('''DROP TABLE sensors''')
        print("connected to db successfully")
    except:
        print("well... almost")
    try:
        conn.execute('''
            CREATE TABLE sensors (
                tstamp TEXT PRIMARY KEY NOT NULL,
                label  TEXT NOT NULL,
                value  REAL NOT NULL,
                unity  TEXT NOT NULL
            );
        ''')

        conn.commit()
        print("sensor table created successfully")
    except:
        print("sensor table creation failed - CREATE TABLE sensors")
    finally:
        conn.close()

    for i in initsensors:
        insert_sensor(i)



if __name__ == "__main__":

    app = Flask(__name__)

    @app.route('/api/initdb', methods=['GET'])
    def api_initdb():
        create_db_table()
        return jsonify(get_sensors())

    @app.route('/api/sensors', methods=['GET'])
    def api_get_sensors():
        return jsonify(get_sensors())

    @app.route('/api/sensors/<id>', methods=['GET'])
    def api_get_sensor(id):
        return jsonify(get_sensor_by_id(id))

    @app.route('/api/sensors/add',  methods = ['POST'])
    def api_add_sensor():
        sensor = request.get_json()
        return jsonify(insert_sensor(sensor))


    #app.debug = True
    app.run(host="0.0.0.0")

the "Dockerfile"

We want to create a Docker image with this web app. As mentioned above, all user images are based on a base image. Since our application is written in Python, we will build our own Python image based on Alpine.

The Dockerfile looks like this:

FROM alpine

RUN apk add --update py3-pip

RUN pip3 install -U Flask

WORKDIR /home/

COPY api.py .

EXPOSE 5000

CMD ["python3", "/home/api.py"]

Build the image

Now that we have the Dockerfile, we can build the image. The docker build command does the heavy-lifting of creating a docker image from a Dockerfile.

When you run the docker build command given below, make sure to replace <YOUR_USERNAME> with your username. This username should be the same one you created when registering on Docker Hub.

The docker build command is quite simple - it takes an optional tag name with the -t flag, and the location of the directory containing the Dockerfile - the . indicates the current directory:

$ docker build -t <YOUR_USERNAME>/restapp .

the generated output is something similar to:

 [+] Building 24.7s (10/10) FINISHED                                                                                
 => [internal] load build definition from Dockerfile                                                          0.0s
 => => transferring dockerfile: 273B                                                                          0.0s
 => [internal] load .dockerignore                                                                             0.0s
 => => transferring context: 2B                                                                               0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                              5.6s
 => [auth] library/alpine:pull token for registry-1.docker.io                                                 0.0s
 => [internal] load build context                                                                             0.0s
 => => transferring context: 4.69kB                                                                           0.0s
 => [1/4] FROM docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9  0.9s
 => => resolve docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9  0.0s
 => => sha256:c059bfaa849c4d8e4aecaeb3a10c2d9b3d85f5165c66ad3a4d937758128c4d18 1.47kB / 1.47kB                0.0s
 => => sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 2.82MB / 2.82MB                0.7s
 => => sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300 1.64kB / 1.64kB                0.0s
 => => sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 528B / 528B                    0.0s
 => => extracting sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3                     0.2s
 => [2/4] RUN apk add --update py3-pip                                                                        5.7s
 => [3/4] RUN pip3 install Flask                                                                             11.5s
 => [4/4] COPY api.py /home/                                                                                  0.0s 
 => exporting to image                                                                                        0.9s 
 => => exporting layers                                                                                       0.9s 
 => => writing image sha256:72eeddfd71ac0199ddca4a513c3f7910be2926ac6c8d73b90bb0acc0ba79ab00                  0.0s 
 => => naming to docker.io/library/restapp                                                                    0.0s            

If everything went well, your image should be ready! Run:

$ docker image ls

and see if your image shows.

Run your image

The next step in this section is to run the image and see if it actually works.

$ docker run -p 8888:5000 --name restapp YOUR_USERNAME/restapp
 * Serving Flask app 'api' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.17.0.2:5000/ (Press CTRL+C to quit)

So now we can execute the following commands:

• http://localhost:8888/api/initdb
• http://localhost:8888/api/sensors
• http://localhost:8888/api/sensors/portenta_h7

the sensor's data updater add.py

As an add on to this example we show the content of a simple sensor's updater that could run in a simple sensor.

import requests, time, datetime, random, json

rest_s_url = 'http://localhost:8888/api/sensors/add'

try:

    # get the new data value
    new_s_value = random.randint(1, 1000)
    tstamp = datetime.datetime.now().isoformat()
    theval = {
        "tstamp": tstamp,
        "label": "portenta_h7",
        "value": new_s_value,
        "unity": "mWatts"
    }

    resp = requests.post(rest_s_url,
        data=json.dumps(theval),
        headers={'Content-Type':'application/json'})
    if resp.status_code != 200:
        # This means something went wrong.
        raise ValueError

except ValueError:
    print("Bad reply from PUT")

Clearly the value of rest_s_url will have to be adapted according to the specific set-up, and also the way we get the sensor's value new_s_value will depend on the sensors itself and the corresponding HW…

IMPORTANT: When we run the app we used the parameters -p 8888:5000 these remapped the 5000 port that is inside the docker container to the port 8888 of the hosting machine… more on this ahead

About the data

All the sensors' information will be lost if the container is deleted. To make it persistent, we have two possibilities:

1. First possibility:

   ​​​​docker run --rm -v "$PWD":/home -p 8888:5000 --name restapp restapp
   

   This will use the "$PWD" to "bind mounts" to the host machine filesystem. This way we can control the exact mountpoint on the host. We can use this to persist data, but it’s often used to provide additional data into containers. When working on an application, we can use a bind mount to mount our source code into the container to let it see code changes, respond, and let us see the changes right away.

2. Second possibility:

   ​​​​docker volume create todo-db
   

   and then

   ​​​​docker run --rm -v todo-db:/home -p 8888:5000 --name restapp restapp
   

   in this way we are using a named volume. Think of a named volume as simply a bucket of data. Docker maintains the physical location on the disk and you only need to remember the name of the volume. Every time you use the volume, Docker will make sure the correct data is provided.
   Where is the data physically located?

   ​​​​$ docker volume inspect todo-db
   ​​​​[
   ​​​​{
   ​​​​    "CreatedAt": "2022-02-01T15:08:21Z",
   ​​​​    "Driver": "local",
   ​​​​    "Labels": {},
   ​​​​    "Mountpoint": "/var/lib/docker/volumes/todo-db/_data",
   ​​​​    "Name": "todo-db",
   ​​​​    "Options": {},
   ​​​​    "Scope": "local"
   ​​​​}
   ​​​​]
   

   But there's a caveat: You can't directly see the contents of volumes on Mac and Windows. This occurs because Docker actually runs a Linux VM to be able to containerize, since containerization is a native functionality for Linux but not in these others OSs. So the path that appears is actually the path inside the VM, and not on your host system. The solution is to create an ephemeral container just to view the content:

   ​​​​$ docker run --rm -it -v todo-db:/home alpine /bin/ash
   ​​​​/ # cd home
   ​​​​/home # ls -al
   ​​​​total 24
   ​​​​drwxr-xr-x    2 root     root          4096 Feb  1 15:08 .
   ​​​​drwxr-xr-x    1 root     root          4096 Feb  1 15:42 ..
   ​​​​-rwxr--r--    1 root     root          4795 Feb  1 14:44 api.py
   ​​​​-rw-r--r--    1 root     root          8192 Feb  1 15:08 database.db
   ​​​​/home # 
   

Push your image

Now that you've created and tested your image, you can push it to Docker Hub.

First you have to login to your Docker Cloud account, to do that:

docker login

Enter YOUR_USERNAME and password when prompted.

Now all you have to do is:

docker push YOUR_USERNAME/restapp