# Job completion issue

## Kubernetes Jobs
Kubernetes Jobs are used when you want to create pods that will do a specific task and then exit. The `restartPolicy` for a Job accepts `Never` and `OnFailure`. The `always` restart policy cannot be applied to jobs.
this is a test

## What is the job completion issue?

*tl;dr* https://github.com/kubernetes/kubernetes/issues/25908

Consider a K8s job with two containers one that does the work and another one getting logs/metrics. The main container will complete the work and exit and side container will keep running and job will never run to completion.
**job.yaml**

apiVersion: batch/v1
kind: Job
metadata:
  name: myjob
spec:
  template:
    metadata:
      name: myjob
    spec:
      restartPolicy: OnFailure
      containers:
      - name: main-job
        image: ubuntu
        command: ["sleep",  "20"]
      - name: side-job
        image: nginx:1.14.2
        ports:
        - containerPort: 80

$ kubectl get jobs
NAME    COMPLETIONS   DURATION   AGE
myjob   0/1           3m21s      3m21s

$ kubectl get pods
NAME               READY   STATUS             RESTARTS   AGE
myjob-vvclb        1/2     NotReady           0          4m13s

The job will never run to completion until the sidecar container somehow able to detect the status of main container, This means sidecar needs to be k8s aware.

## Proposal:

A possible solution could be one container can be marked as a main container, and if it exit all other containers should follow up e.g If a job have one main container and other one is logging container the job should gets completed if the main container is exited.

We can assign one container in a pod to be called as main container, via an **annotation**. If that container exits, other containers should be killed by the kubelet and terminates the pod.

apiVersion: batch/v1
kind: Job
metadata:
  name: myjob
spec:
  template:
    metadata:
      name: myjob
      # adding annotation name of the main container to pod
      annotations:
        maincontainer: "main-job"
    spec:
      restartPolicy: OnFailure
      containers:
      - name: main-job
        image: ubuntu
        command: ["sleep",  "20"]
      - name: side-job
        image: nginx:1.14.2
        ports:
        - containerPort: 80

- When the main container of a job is exited successfully(exitCode 0), Then kill the other container.
- When the main container exited unsuccessfully

    If restart policy==never
        Kill the sidecar and mark the job as unsuccessful
    If restart policy == onFailure
       Then do nothing on the sidecar container.

Managed to get it working with the small code snippet https://github.com/adisky/kubernetes/pull/1/files, without considering any edge cases :)

$ kubectl get jobs
NAME    COMPLETIONS   DURATION   AGE
myjob   1/1           24s        24s

$ kubectl get pods
NAME           READY   STATUS             RESTARTS   AGE
myjob-hw4jw    0/2     Completed          0          35s

## Some questions

1. This solution can only be applied when the restartPolicy is `never` or `onFailure`, for restart policy `always`. kubelet will always try to restart pod so we can not kill any other containers.
2. This has to applied on `Jobs` only or other types of workload as well, need to evaluate on this.
3. one main container or there could be multiple main container? need to evaluate on the cases of multiple container.
4. Need to evaluate all the probe behaviour(liveness, readiness, startup) with it.
5. When killing the pod, how to ensure we keep sidecars running until main is terminated? (I think we could modify the kill to focus only on main container, and let your code change handle others, probably will handle probes as well)
6. Same question for startup? (I realize we might already touch the ordering bit... but that's gonna be the first question from istio)

As comment from here https://github.com/kubernetes/enhancements/pull/1980#issuecomment-698481683
keystone/critical containers:
If multiple keystone containers and one completes(exit code 0)
-- wait for all keystone containers to complete
-- if all are complete then kill the sidecars
If multiple keystone containers and one stops (exit code Non zero)

This can be implemented in Job controller also, If the keystone containers are not completed controller can request pod delete and mark the job as completed.?????

how to relate it to controllers other than job if like deployment?
we are killing side car containers if the keystone containers are completed (exit code). It won't affect deployment or other controller
restart

Think about the case where we add restart policy per container?