How to Build a Task System using Go Docker SDK?

Go is designed to enable developers to rapidly develop scalable and secure web applications. Go ships with an easy to use, secure and performant web server and includes its own web templating library. For enterprises, Go is preferred for providing rapid cross-platform deployment. With its goroutines, native compilation, and the URI-based package namespacing, Go code compiles to a single, small binary—with zero dependencies—making it very fast.

Developers love Go as it performs better because of its concurrency model and CPU scalability. Whenever developers need to process some internal request, they do it with separate Goroutines which are 10x cheaper in resources than Python Threads. Using static linking, Go actually combines all dependency libraries and modules into one single binary file based on OS type and architecture.

Table of Contents

• How to Build a Task System using Go Docker SDK?
  • Table of Contents
  • Building the Application
  • Key Components
  • Getting Started
    • Define Task
    • Task Runner
    • Container Manager
    • Entrypoint
  • Steps to Run
  • Sequence Diagram
  • Conclusion
  • References

Building the Application

In this blog tutorial, we will learn how to build a basic task system (Gopher) using Go.

First, we’ll create a system in Go that uses Docker to run the tasks. Next, we’ll build a Docker image for the application. The goal here is to give you an example how you can use Docker SDK to build some cool projects. Let’s get started.

Key Components

• Go
• Go Docker SDK
• Microsoft Visual Studio Code
• Docker Desktop

Getting Started

Once you have Go installed on your system, follow these steps to build a basic task system using Docker SDK.
Here is the directory structure that we will have at the end:

➜ tree gopher     
gopher
├── go.mod
├── go.sum
├── internal
│   ├── container-manager
│   │   └── container_manager.go
│   ├── task-runner
│   │   └── runner.go
│   └── types
│       └── task.go
├── main.go
└── task.yaml

4 directories, 7 files

The complete code for the task system is available at dockersample/gopher-task-system.

Define Task

First and foremost we need to define the structure of our task. Our task is going to be a YAML definition which takes the following structure:

version: v0.0.1
tasks:
  - name: hello-gopher
    runner: busybox
    command: ["echo", "Hello, Gopher!"]
    cleanup: false
  - name: gopher-loops
    runner: busybox
    command:
      [
        "sh",
        "-c",
        "for i in `seq 0 5`; do echo 'gopher is working'; sleep 1; done",
      ]
    cleanup: false

The following table describes the task definition:

Key	Description
version	The API version (we are not really using it)
tasks	A list of tasks
task.name	Name of the task
task.runner	The Docker image a task uses to run
task.command	A list of string; an argument to the runner
task.cleanup	If true, the task container will be removed after completion

Now that we have a task definition, let's create equivalent Go structs.

// internal/types/task.go

package types

// TaskDefinition represents a task definition document.
type TaskDefinition struct {
	Version string `yaml:"version,omitempty"`
	Tasks   []Task `yaml:"tasks,omitempty"`
}

// Task provides a task definition for gopher.
type Task struct {
	Name    string   `yaml:"name,omitempty"`
	Runner  string   `yaml:"runner,omitempty"`
	Command []string `yaml:"command,omitempty"`
	Cleanup bool     `yaml:"cleanup,omitempty"`
}

Task Runner

The next thing we need is a component that can run the tasks for us. We will call it Runner and here is how we define it:

// internal/task-runner/runner.go

type Runner interface {
	Run(ctx context.Context, doneCh chan<- bool)
}

Note that we are using a done channel (doneCh). It is required because we would like to run our task asynchronously and once the task is complete, we would like it to tell us so.

The complete definition of the task runner can be found here. We would only talk about specific pieces of the code here:

// internal/task-runner/runner.go

func NewRunner(def types.TaskDefinition) (Runner, error) {
	client, err := initDockerClient()
	if err != nil {
		return nil, err
	}

	return &runner{
		def:              def,
		containerManager: cm.NewContainerManager(client),
	}, nil
}

func initDockerClient() (cm.DockerClient, error) {
	cli, err := client.NewClientWithOpts(client.FromEnv)
	if err != nil {
		return nil, err
	}

	return cli, nil
}

The NewRunner returns an instance of the struct providing the implementation of the Runner interface. The instance will also hold a connection to the Docker engine. This connection is initialized in the initDockerClient function by creating a Docker API client instance from environment variables.

By default it creates an HTTP connection over Unix socket unix:///var/run/docker.sock (the default Docker host). If you would like to change the host, you can do so by setting the DOCKER_HOST environment variable. The FromEnv will read the environment variable and do the needful.

The Run function, as defined below, is fairly simple. It loops over a list of tasks and executes them. It uses a channel named taskDoneCh, to know when a task is done. It's important to check whether or not we have received a done signal from all the tasks, before we return from this function.

// internal/task-runner/runner.go

func (r *runner) Run(ctx context.Context, doneCh chan<- bool) {
	taskDoneCh := make(chan bool)
	for _, task := range r.def.Tasks {
		go r.run(ctx, task, taskDoneCh)
	}

	taskCompleted := 0
	for {
		if <-taskDoneCh {
			taskCompleted++
		}

		if taskCompleted == len(r.def.Tasks) {
			doneCh <- true
			return
		}
	}
}

func (r *runner) run(ctx context.Context, task types.Task, taskDoneCh chan<- bool) {
	defer func() {
		taskDoneCh <- true
	}()

	fmt.Println("preparing task - ", task.Name)
	if err := r.containerManager.PullImage(ctx, task.Runner); err != nil {
		fmt.Println(err)
		return
	}

	id, err := r.containerManager.CreateContainer(ctx, task)
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println("starting task - ", task.Name)
	err = r.containerManager.StartContainer(ctx, id)
	if err != nil {
		fmt.Println(err)
		return
	}

	statusSuccess, err := r.containerManager.WaitForContainer(ctx, id)
	if err != nil {
		fmt.Println(err)
		return
	}

	if statusSuccess {
		fmt.Println("completed task - ", task.Name)

		// cleanup by removing the task container
		if task.Cleanup {
			fmt.Println("cleanup task - ", task.Name)
			err = r.containerManager.RemoveContainer(ctx, id)
			if err != nil {
				fmt.Println(err)
			}
		}
	} else {
		fmt.Println("failed task - ", task.Name)
	}
}

The internal run function does the heavy lifting for the runner. It accepts a task and transforms it into a Docker container. It uses a ContainerManager to execute a task in the form of a Docker container.

Container Manager

The container manager is responsible for:

• pulling Docker image for task
• creating the task container
• starting the task container
• waiting for the container to complete
• removing the container, if required

Therefore, with respect to Go, we can define the container manager as:

// internal/container-manager/container_manager.go

type ContainerManager interface {
	PullImage(ctx context.Context, image string) error
	CreateContainer(ctx context.Context, task types.Task) (string, error)
	StartContainer(ctx context.Context, id string) error
	WaitForContainer(ctx context.Context, id string) (bool, error)
	RemoveContainer(ctx context.Context, id string) error
}

We have the containerManager which provides the implementation for ContainerManager interface:

type DockerClient interface {
	client.ImageAPIClient
	client.ContainerAPIClient
}

type containermanager struct {
	cli DockerClient
}

Notice that containerManager has a field called cli of type DockerClient interface, which itself embeds two interfaces from the Docker API - ImageAPIClient, and ContainerAPIClient. Why do need that?

For the container manager to perform its operation it needs to act as a client for the Docker engine/API. And the client will work with Images and Containers, it must be of a type that can provide the required APIs. Hence, we need to embed the core Docker API interfaces and create a new one for us.

The initDockerClient function (seen above in runner.go) returns an instance that implements the required interfaces. Hence, everything put together works the magic. You can the documentation here to better understand what is returned when we create a Docker client.

The complete definition of the container manager can be found here.

Note: We have not individually covered all functions of container manager here, otherwise the blog will be too extensive.

Entrypoint

Now that we have the individual components covered, let's put all the pieces together in the main.go which would be our entrypoint:

// main.go

package main

func main() {
	args := os.Args[1:]

	if len(args) < 2 || args[0] != argRun {
		fmt.Println(helpMessage)
		return
	}

	// read the task definition file
	def, err := readTaskDefinition(args[1])
	if err != nil {
		fmt.Printf(errReadTaskDef, err)
	}

	// create a task runner for the task definition
	ctx := context.Background()
	runner, err := taskrunner.NewRunner(def)
	if err != nil {
		fmt.Printf(errNewRunner, err)
	}

	doneCh := make(chan bool)
	go runner.Run(ctx, doneCh)

	<-doneCh
}

Here is what we do:

• Validate arguments
• Read the task definition
• Initialize task runner which in turn initializes container manager
• Create a done channel receive the final signal from runner
• Run the tasks

Steps to Run

• Clone the repository

git clone https://github.com/dockersamples/gopher-task-system.git

• Build the task system

go build -o gopher

• Run tasks

$ ./gopher run task.yaml

preparing task -  gopher-loops
preparing task -  hello-gopher
starting task -  gopher-loops
starting task -  hello-gopher
completed task -  hello-gopher
completed task -  gopher-loops

• Run docker ps -a to see what all task containers

$ docker ps -a
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS                     PORTS     NAMES
396e25d3cea8   busybox   "sh -c 'for i in `se…"   6 minutes ago   Exited (0) 6 minutes ago             gopher-loops
aba428b48a0c   busybox   "echo 'Hello, Gopher…"   6 minutes ago   Exited (0) 6 minutes ago             hello-gopher

Note that in task.yaml the cleanup flag is set to false for both the tasks. It was done on purpose to get containers list after task completion. Set it to true and the task containers will be removed automatically.

Sequence Diagram

Conclusion

In this blog post, we have built a simple task system using Docker SDK for Go. Docker is mostly used as an Ops platform to run workloads. However, with Docker SDK you can build cool tools of your own which eventually helps you better understand how Docker works under the hood. We look forward to sharing more such examples and cover different things you can do with Docker SDK.

References

• Docker SDK
• Go SDK Reference