# Developer Tooling

What we have so far?

## CI
### What We Have

- All tests are currently run
- Unit Test
- Integration tests
- End to end tests
- E2E is run on _all_ platforms (MacOS, Windows, Linux)
- No Test Covergae
- End to End/Integration testing looks like :
- Build JITX
- Install the build
- Run all integration/tests
- Currently sequential
- Parallelize these things
- smoke tests
- Other Tests split into two parts :
- Unit tests that can be compiled/run indepedently
- Integration tests that require compiling the "JITX plugin" that requires compiling a few targets besides the tests
- CI on PRs is 30-40 minutes
- Releases
- Tag release candidate on develop branch
- Triggers release infrastructure
- Build & Test
- Spit test back up to

## What we want
### Goals
- Fewer developer interrupts when things go right
- Better output/tools when tests fail
- Speed of everything
- Code coverage
- Quicker PR turnaround times
- Automated changelog
- Handle Breaking Changes
- One giant integration test that covers the entire public API
- If not in the integration test, not part of the API

### Examples
- Break out into fewer integration tests per PR
- More parallelism
- Run integration tests on merge
- Code coverage tool!
- Only run relevent tests
- Run test suites based on modifications
- Dependency tracking between tests
- Bring down CI times!
- Sequencing of PRs is not well defined
- Ideal world where hosting costs is negligible :
- There is a full set of integration, end to end, unit tests run on every commit. Takes hours.
- At any point of time, see where we're at. See some set of tests.
- In meantime do some unit tests
- Feel good about the PR, merge to staging branch.
- Bot that runs CI then on staging branch on merge of develop/staging branch. Auto merge to develop when passes.
- Way to visualize test results
- Feature flags are good
- Each time merge into develop, run CI on develop.
- Notifications :
- When your run fails, ping you on slack/email
- When nightly fails notify
- Better incremental compilation
- Stanza has incremental compilation
- Verify numbers, see how much we can improve
- Talk to Scott, Jonathon about this
- Minimize developer interruptions
- GTest for Stanza!

## Questions

- Do we want to have a staging branch
- Add things into staging branch w/o CI passing, or partial CI passes
- Only run integration tests on staging branches
- Perhaps start using feature branches
- Branch for each sprint?
- Merge into branches into
- Releases and feature flags?
- Release process to cut a release candidate
---