# Project Management Flow (Concept)

###### tags: `Practices`

## Introduction

The current document describes the (proposed) flow of how the Exartx team handles the triage and execution of the tickets in the backlog.

## Overview

Enarx is a project that leverages GitHub for all aspects of development, review and automation. The primary way to make changes to the project is by submitting a Pull Request (PR). The team tracks the work via issues and PRs. Developers mostly look at those issues and PRs and don't see value in other project management tools. In this given situation the introduction of additional project management tools seems artificial. The closer and the more natural the process and flow for the development team is - the better.

GitHub offers a feature called Projects. Projects allow a flexible view of the tickets in tabular and board modes.

## Requirements for Project Management Tools and Flow

The process and tools chosen for the management of the Enarx project shall address the following functional and non-functional requirements:

* **Usability**
* The process should be simple and intuitive. If only a project manager understood how to run the project, it would be considered a failure. Any engineer on a team should not only understand the process, but it should be logical and natural and thus not require a lot of training.
* Tools and processes should support the flow described in the contribution guide and enable it rather than be an obstacle.
* Tools and processes should allow the team to focus on specific tasks, but the less important things should get in the team's way. The project should track the issues but not clubber the view if they are irrelevant or unimportant.
* Any team member should be able to use tools and flow to run the Enarx status meeting without much preparation or training.

* **Functional**
* The tools and process shall allow seeing clearly what the team is currently working on, and what is in the immediate potential pool of work.
* The tools and process shall allow effective (non-time consuming) triage and planning processes so that the status meetings are short.
* Team shall always know and understand what to work on next.
* It should be possible to gradually move from a non-iterative flow to a more iterative and incremental workflow so that the work planned for a sprint renders commits nearly daily.
* Triage process must be well-defined and intuitive.
* Ticket management for the tickets in development must be lightweight and not create extra bureaucracy for developers.
* The set of tickets the team works on at a time should be minimal and be limited to up to three tickets per person.
* There must be a 2-3 day supply of available work for every team member to pick up if they get blocked. This supply, however, must be at most three days of work per person.

## Reality Check

In reality, the process is quite complex. The overall problem space consists of more than one GitHub organization, and the team would need to track the efforts across all of them.
Within the Enarx organization, multiple streams of work are happening in parallel and done by different people. For example, Chores and Maintenence work, Kernel work, development of the specific crates, WebAssembley capabilities to support different types of workloads, sample applications for try.enarx.com and many others.
The tasks and features sometimes take significant effort to size.
Usually, the work consists of a series of PRs. PRs go upstream for review by community members. While a PR is in the queue, the developer can switch to another feature or task.

## Flow (draft)
### Core Principles

* Focus on the developer, not the manager. Make it simple and natural for developers to do their job.
* Keep things as simple as possible. Try to solve only some of the use cases from the beginning. Focus on the main ones and make them work.
* Allow the process and tools to evolve as habits change, the team grows, and the project matures.

### Main scenarios

The development flow includes the following main situations:
* **Creation** when a PR or issue is filed by a team or a community member.
* **Triage** when a PR or issue is reviewed and planned for execution.
* **Execution** when the issue is worked on. This effort includes:
* Development of code or documentation (For example, design documentation or manual).
* Creation, integration and execution of the related tests.
* Review the work by peers or other community members as required by corresponding community projects.
* Acceptance of the work, which leads to the work item officially being merged, tested and thus marked as done.

> Note: A more detailed "definition of done" will be discussed separately.

### Flow overview
* Issues and PRs are created in the repositories they belong to. In most cases, developers open them to track future work. Any new issues or PRs should be automatically tagged as **Status:New** via workflow actions.
* New issues and PRs need to be triaged unless they were addressed immediately and have already been merged and closed by the time of the Status meeting the next day.
* During the triage, only issues that need to be triaged should be in focus. making it simple to classify the issue according to the **Effort** classification, and identify the **Priority**.
* Triage is a simple process of answering the following questions:
* **What is this issue about?** - an answer to this question shall render the **Effort** that this issue falls under. See the list of the (proposed) efforts later in this section.
* **How important is this issue?** - an answer to this question shall render the **Priority** setting following the classification defined in a section below.
* **When should we work on it?** - the answer to this question would help to decide how the issue affects current efforts. See the section about "Scheduling Work".
* **Who should work on it?** - if this is urgent or high-priority work, the assignee must be identified. One of the sections below covers the factors to consider while assigning the work.

#### Efforts Classification

The following classification of the efforts is suggested based on the current (***November 2022***) backlog review.

* **Chores and Maintenance** - CI, infra, ownership, same workflows and policies
* **Kernel** - all the Kernel efforts including bugs, support of new HW or use cases
* **WebAssembly features** - VFS, threads, async, sockets etc.
* **Special crates** - additional crates
* **Examples** - sample applications for try.exanrx.com
* **Attestation (Stewart)** - all the aspects of attestation and signing
* **Registry (Drawbridge)** - all the aspect of customer workload management
* **Documentation** - web site community documentation
* **General maintenance** - fixing issues and making minor enhancements in otherwise well-working code.

The list of **Efforts** can be refined and extended as needed. Some of the efforts might be further detailed and broken into smaller efforts. When the effort is complete it can be retired and not used anymore. In such cases, follow-up bugs and issues will be placed under the **General maintenance** bucket.

#### Priority
The following priorities are chosen from during the triage:
* **Urgent** - this is a blocking issue and shall be addressed immediately by someone on the team at the expense of some other planned work.
* **High** - this is an issue of high importance. The team should start working on this issue relatively soon - in 1-2 days. The team should make a concerted decision to avoid using “High” for a significant number of tickets, thus creating bottlenecks and piling too much work on their own plates.
* **Medium** - work that will be done as the team get cycles to do it.
* **Low** - nice to have work that might be done at some point if the team gets extra help or its importance would grow due to changing circumstances.
The priority of an issue can evolve based on the circumstances and situation. But the general expectation is that the team works on **Urgent** and **High** priority issues and turns to **Mediums** only when there is not enough supply of the **High** priority ones.
**Low** priority items are not expected to be worked on unless there is extra help. **Low** priority items are subject to periodic (quarterly) pruning.

#### Scheduling Work
Scheduling the work is indicated by changing the **Status** of a ticket to "Scheduled". Other statuses are:
* **New** - issues before the triage have this status.
* **Backlog** - issues that the team identified during the triage as not urgent to work on.
* **Scheduled** - issues that need to be worked on within a couple of days or immediately.
* **In Progress** - issues that are being executed following the execution flow.
* **Done** - all the activities related to an issue are completed. If there is any deferred outstanding work, it needs to be captured in a new ticket.

Work can be scheduled (or not scheduled) using the following logic:
* If the issue is **Urgent** it should be scheduled immediately. An assignee needs to be appointed to it. Some other work of this assignee needs to be deferred or moved to someone else causing that other's person's queue to grow (within the declared limit) or for the person to defer some of their work. The work is scheduled by placing it into the **Scheduled** bucket
* If the issue is triaged as **High** and there is capacity in the **Scheduled** bucket, then it can be added. Otherwise, it has to either bump something or go into the **Backlog**.
* If the issue is **Medium** or **Low**, it goes into the **Backlog**. It can be **Scheduled** only in the absence of the more important work or if external help is offered.

#### Assignee
The assignee must be chosen if the work is being scheduled, i.e. if someone plans to start working on it immediately or within a couple of days. No **Scheduled** work is allowed to be unassigned. In the future, some automation may be added to enforce that.

#### Triage and Planning
With the daily standups, the team effectively runs one-day-long iterations (or no iterations at all). This is an artifact of the complexity of the situation when there are too many fires that need to be dealt with simultaneously by the small team. An organized and rigorous planning, together with the issue decomposition, would help to get out of this mode and move the team to a more predictable cadence of 1-2 week-long iterations.
> Note: The details of this process will be augmented as the project management practice matures. For now, we assume daily review and triage together with the absence of the sprint planning activities.

#### Pruning
Once a quarter Status meeting will be dedicated to pruning. In such meetings **Low** and some **Medium** priority items will be re-reviewed and closed, keeping the backlog more lean and clean.

### Schematic View of the Flow

flowchart 
    classDef withmargines fill-opacity:0.0,color:#FFFFFF,stroke-width:0px;
    subgraph Creation
        person((User))
        issue(New Issue)
        pr(New PR)
    end
    
    subgraph Triage
        effort{Which effort?}
        priority{What priority?}
        work{Should we work\non it\nnow/soon?}
    end
    assignement{{Assigned}}
    subgraph Scheduling
        foo1["<p style='width:0px;height:100px;margin:0'>Invisible !!!/p>"]:::withmargines
        backlog(Backlog)
        scheduled(Scheduled)
    end
    subgraph Execution
        foo2["<p style='width:0px;height:100px;margin:0'>Invisible !!!/p>"]:::withmargines
        inprogress("In Progress")
    end
    done{{"Done"}}
    
    person--"Opens"-->issue
    person--"Opens"-->pr
    issue-->effort
    pr-->effort
    effort--"Effort classification"-->priority
    priority--"Prioritization: Low/Medium/High/Urgent"-->work
    work--"Yes => usually priority is #quot;Urgent/High#quot;"-->assignement-->scheduled 
    work--"No => usually priority is #quot;Low/Medium#quot;"-->backlog
    scheduled--"Picked up by a developer.\nDeveloper changes the status."-->inprogress
    inprogress--"1-2 days"-->inprogress
    inprogress-->done
    

### Tooling
GitHub Projects seems to be the best way to meet the requirements and achieve the outlined goals while following declared principles.

#### Project
The GitHub Project for Enarx can be found [here](https://github.com/orgs/enarx/projects/10). It is a public project that includes multiple views that allow triage and planning of the work with the Enarx project organization.
> Note: A similar Project will be created to handle Drawbridge and Steward projects if this proposal is apporved.

#### Naming convention
* There are two types of the views that GitHub Projects feature supports: Table and Board. There is an icon that distinguishes one from another, however, for easier differentiation, the names of the boards follow the following naming convention: "Board: *name of the board*".
* Table views usually allow grouping. To show that the table view supports grouping of issues by specific fields put the grouping field into the parentheses like this: "*name of the view (grouping field)*", for example, "Main triage view (Status)" will present the view useful for the triage with the tickets grouped by the status.

#### Automations
Currently there is single automation that is defined for tickets and PRs: the newly created tickets and PRs are automatically assigned "New" status.
Other automation is possible in the future. For example:
* Assigning **Backlog** status as soon as the ticket is triaged as **Medium** or **Low**.
* Assigning **Scheduled** status as soon as the issue is triaged as **Urgent** and the person is assigned to the issue. However, this automation should consider capacity planning to avoid creating too many tickets in **Scheduled** or **In Progress** status at the same time overwhelming the team. To effectively do this, we can start limiting the number of tickets per person within the **Scheduled** and **In Progress** buckets, for example, making the hard limit of three tickets/PRs per person. In the future, a more fine-tuned approach might be implemented based on the story points.
Some of the inspirational ideas on how to implement the automation can be taken from [here](https://github.com/marketplace/actions/github-project-automation).

#### Flow with the tool
* The [tiage view](https://github.com/orgs/enarx/projects/10/views/5) is used to conduct the triage phase as shown on the flow chart.
* The [tiage view](https://github.com/orgs/enarx/projects/10/views/5) can also be used for assignment and scheduling however other offered views might also provide a convenient way for scheduling.
* The [Main](https://github.com/orgs/enarx/projects/10/views/2) and the [High/Urgent](https://github.com/orgs/enarx/projects/10/views/11) boards provide a convenient way to see what is scheduled, what is in progress and to start working on something as well as designate work as "Done". Those boards are also intended for making trade-off decisions but they might need a further tune-up for this use case.

#### Known Issues
* It looks like the project has be linked to all the repositories manually. It has been done for all existibnf repositories, but if a new repostory is added, if should be added manually.
* The search for new issues to add is not convenient. One has to add it per repository. It might be OK when we triage and review the rest of the backlog. This might need to be further automated.
* When there are no tickets in the "No Efforts" category, the tickets are added with the Effort that is already cpecified. This might lead to creation of a special board just to add the tickets.