# Pandemia Snark Hunt

## AS Notes

1. How do you feel, looking back on the project?
- Would you do it again?
- Yes
- Has it changed your life?
- Not particularly.
- What was your lowest point?
- Writing the end-to-end tests was fustrating because of the length of time it took for each run.
- What was your best time?
- The satifaction expressed by the researcher when all the strands of documentation where pulled together.

2. Would the project have been better if it had been one-third the length? Three
times the length?
- With only one month it would be hard to achieve anything worthwhile.
- With nine months, we could have done a lot more in-depthdefactoring and rigorious testing. We would have been reasonable to work on some new features including some of the ideas around machine learning, policy optimisation.

3. Someone else is about to kick off a very similar project. We'd like the outcome to be *ten times better* than this one. What do you tell them?
Wrong question - Before the "about to kick off" point - there should be a risk register as part of the project apprisal process (note there may need to be seperate sections/registers for "the project as a whole" and "REG's involvement in it").
However - If they are starting tomorrow:
- be clear and realistic about your expectations
If they have a month to prepare:
- have a better toolset/workflow for cross-platform development and testing
- Have some set of discussions between the REG team about expectations of the project, what they individally might be able to contribute to it, and how they might work as a team. Also what weaknesses/skills gaps they might lack and hence any assistance that might be needed. (This might need to be a facilitated discussion, to avoid the tendancy for start-of-project-over-optimition).
Important to note that; Aoife and I hadn't worked together before hand. This was *not* a problem, has I felt our different skills complenmented each other, but it is good to be prepared.

4. What do you wish you had done differently?
- Could you have forseen this issue?
- Really?
- Could you have done anything different even if you had forseen it?
- What were the barriers to doing something different?
- Could /they/ have been removed?
- Really?

Quoting from an email exchange on this:
> Overall we have done a lot of useful stuff, and the code and docs are in a better place as a result. What we have done less of is coach James on how to ensure that the development of pandemia continues on a sustainable basis. Things we might be covered would be:
>
> - The software development techniques that have been missing so far.
> - More significantly, the project and product management viewpoint needed to get peers engaged and invested in the success of the tool (not just funding).
>
> Realistically, I feel that this would have been difficult to do much more on this within the time scale. It has taken until now to understand the project's current state, history and ambitions and build the level of trust required. If I had to pick the thing I am most disappointed with about the project, it would be this.

5. What will the impact of this project be ten years from now?
- A goal should also be what if any significant impact *within REG* one year from now? How would we achieve that, and what are the barhiers?
- If there is any significant impact from this project in ten years time it will be from advocating and demostrating the benefits of disaplined research software engineering, rather than directly from then lines of code we wrote.

6. What else should go in the Snark Hunt Review?

I think we need to consider what we want to achieve with the Snark Hunt, but perhaps that's a discussion for the away day.

Pending that I suggest focusing on the chronology of the project, eg:

- Inception
- proposed REG involvement and project apprisal process
- Kick off
- Project rythum and mid-point
- Ending research/results/outputs with colaborators
- Wrapping up withing REG - archiving/reporting

What are the leasons (good and bad) to be learnt at each stage? What actions (including changes to documented and undocumented processes) would help us repeat/aviod then in the future.

* "more training on topic X" is often valid, but also too easy an answer and hence should be scrutanised.
* Some things are so rare, or project specific, that it is OK to simply note them without any proposed action.

We used the Definition-of-Done from the REG-Handbook to assess the software at the beginning and end of the project. It is a crude and subjective measure, but none the less I think it was a useful tool:

| | **Before** | **After** |
| --- | --- | --- |
| **Reach** | Level-01 | Level 3-4 |
| **Robustness** | Level-01 | Level 2-3 |
| **Functionality** | Level ? (did not assess) | Level 3-4 |
| **Documentation** | Level 2 | Level 4 |
| **Testing** | Level 1 | Level 3 |
| **Open Release** | Level 1-2 | Level 3 |

-- End Andy's notes --

## AH Notes

### Example answers to Q's below

1. Overall feelings on project?
- **Would I do it again?**
- Maybe.
- Would need a few conditions to be met:
- Longer project time
- If this was my main project
- **Has it changed my life?**
- I hope not
- **What was the lowest point?**
- Realising the state of the project
- Think this happened a few times
- Initially, midpoint when we decided to scale back expectations, towards the end of project when we just got to grips with things
- **What was your best time**
- Getting stuck into some good old-fashioned C code and thinking about memory
2. Would the project have been better if it had been three-times the length?
- Yes
3. Someone else is about to kick off a very similar project. We'd like the outcome to be *ten times better* than this one. What do you tell them?
- When a project is very focused on an existing codebase like this, perhaps having some time to clone the repo and do a blind dive in
- This might help direct the initial scope phase
- _However_, I do not believe this project would have been accepted if this was done.
- I'd be super interested if someone wanted to try scoping this project based purely on the last commit pre-REG joining.
- Perhaps suggest a co-working session with the collaborator
- We had a lot of variables named like `sigma_f11abc=0.1` and the explanation of this was locked in the collaborator's head
- At one point I asked "X, if you got hit by a bus tomorrow, who else could we ask to explain this variable's meaning?" To which the answer was no-one.
- This worked surprisingly well, see misc points, and the collaborator to their credit did make steps to address this issue and worked with us and our subject ignorance excellently.
4. What do you wish you had done differently?
- **Could you have forseen this issue?**
- **Could you have done anything different even if you had forseen it?**
- **What were the barriers to doing something different?**
- **Could /they/ have been removed?**
5. What will the impact of this project be ten years from now?

### Misc Points
- Feel it's important to make a mention of how helpful Sam M was at finding an obscure C int type bug
- For ref, long longs in windows != with unix. And doubly confused by passing numpy arrays <-> with C.
- Which brings me to next point
- It would be helpful to evaluate workflows and how easy they are for us to work with
- e.g., This code was originally running only on Windows. Debugging to the degree we had to, and the long run times of models are poorly suited to cloud solutions.
- I had to use my personal VM solutions for Windows (paid Parallels mac subscription).
- It might be nice for REG to have some standard protocols for this, e.g., if we require a Windows machine to test/debug evauluate a `typical` user's experience, what is the goto? if this comes up on a future project, what *should* REG people do?
- I think we should use the [Bus Factor](https://en.wikipedia.org/wiki/Bus_factor) in a few situations
- What is the N number of people required to work on this project
- e.g., what if between signing on for REG and starting the work, someone key leaves the project.
- Can our work progress if we depend on knowledge from 1 person who may be difficult to pin down for meetings
- Do we communicate and document these communications in such a way that it is accessible readily and easily
- .e.g., here we depended on a mix of teams, github, emails and slack (which the collaborator couldn't access) If one of us (Andy, or myself) dropped off the project and someone else took over, can they gather the information required easily?
- Some kind of musing about the nature of REG
- Which turns into two questions
1. What does REG want
2. What does Turing want (from reg)
- For example, **is REG happy with this project**
- if why / why not?
- if unclear then see above
- How can REG quantify if this was a good project to take independantly from the people working on it

## Inteviewer Potential questions

The following is a draft list of questions by the faciliator of the
participants. The sub-bullets are ideas for follow-up.

1. How do you feel, looking back on the project?
- Would you do it again?
- Has it changed your life?
- What was your lowest point?
- What was your best time?

2. Would the project have been better if it had been one-third the length? Three
times the length?

3. Someone else is about to kick off a very similar project. We'd like the outcome to be *ten times better* than this one. What do you tell them?

4. What do you wish you had done differently?
- Could you have forseen this issue?
- Really?
- Could you have done anything different even if you had forseen it?
- What were the barriers to doing something different?
- Could /they/ have been removed?
- Really?

5. What will the impact of this project be ten years from now?

6. What else should go in the Snark Hunt Review?