# GitOps repo structure

I don’t know that this can or even should be standardized. I think repo organization Is subject to Conway’s law, especially with GitOps because your source repo is now an integral part of your overall system.
- manage up
- manage down

However, I do think we can provide some good questions to ask, good rules of thumb, and common patterns that match common use cases for those just getting started or looking to iterate on their current approach.

Proposed rules of thumb (axioms, not best practices per se):
1. There should be no standard for GitOps repo structure
2. GitOps tooling should be flexible enough to allow for iterating on repo structure
- It is better to get started and iterate flexibly, than to either wait or adopt a static model however perfect it may promise to be
- Choose GitOps tooling and techniques that is flexible enough to play well with this iterative approach
3. Repo structure should be relatively easy to administer for those whose job it is to set it up
4. Repo structure should be as clear as possible for developers to understand and engage with according to their level of access to and responsibility for the system

Needs change over time, and the journey of GitOps can even help you discover more of all your needs and use cases as the velocity and flexibility of actual your projects increase with CD. So don’t get stuck on convention, instead seek configurable tools and techniques that help you maintain flexibility.

However we can’t just leave you off with a relativist answer like this. BC there are some known patterns that have worked well for different people and organizations over time so far. We’ll present some of those to you now, with clear examples, matching use cases, and note some pros and cons of each. So that you can decide for yourself what your next best step is in your gitops journey, based on where you’re at now and on your immediate known needs, pain points, value proposition of the known solutions, and your organization’s overall goals.

## Demo:
mention for simplicity:
- For this demo, using PLAIN YAML ONLY to start
- "How do I manage all this YAML?" This is a separate talk (we have many talks and demos on ways to do this)
- explain that other tools like kustomize, Helm, Terraform are tool to help keep your manifests DRY. They do other things too, but this is what they have in common for k8s: they are popular formats for k8s manifests with less repeating YAML

1 part showing off, 1 part reassurance:
- Uptime and ease: How does it work on a running system? Let's find out! (spoiler: it won't break your running systems, and you don't need to jump through flaming hoops to do it)
- Auditing: Git history. Remember Git has built-in tools to allow structural changes - even for splitting into or merging multiple repos - while still retaining a full audit trail. Learn and use these according to your internal or external audit needs. Suggestion: default to keeping a clear Git history in some repo or branch, because you can always squash history later but once erased completely those options are gone forever

Say something about Flux building blocks

## Patterns:
- monorepo
- Repo per environment
- Repo per team
- Repo per app

## Considerations:
- current and ideal team/org structure
- Visibility needs per role/team (including sensitive content restrictions, and auditing) - READ ACCESS
- access control needs and gates (who should be able to change what, with whose approval) - WRITE ACCESS
- Simplicity and ease of use for your users (platform engineers, delivery automation engineers, app developers, auditors)

## Demo
(goal: script out and keep as SIMPLE as possible to show the point):
- How to start with monorepo with Git commits, and break into smaller repos retaining only commits for those directory's files
- How to have multiple repos, then merge into a shared repo – also retaining correct commit history
- How to do this safely in either direction (from wide to narrow or reverse) on a running GitOps system
- start with one monorepo with one kustomize dir per environment
- follow along with breaking that repo into repo per environment
- you can also move directly from monorepo to repo team or per app, by pausing kustomization and splitting into new kustomizations for the desired level of specificity, then enabling all those (no changes should happen on your cluster because manifests are all the same, just can now pause per team/app), then deleting the earlier paused mono-kusstomization just by removing it's manifest
- you can also do the reverse of this, to match the merging of repos from more narrow, separate ones – by pausing all the individual kustomzations you want to group together, then making one to share them all in the new combined repo, then deleting the old one simply by removing all the old declarations
- You can back out of any of these processes – in either direction – by reversing the steps. As long as git history is retained on some branch in all repos, all of this is acheievable reardless of your internal or external audit requirements, and regardless of your gates for specific apps, teams, or environments

Christian Hernandez:
- He ALWAYS starts this conversation with Conway's Law, so agrees this is helpful
- He's writing a book chapter on this topic right now! Though he is approaching it from a different angle. He likes our angle, and thinks this would be a good approach for a document for OpenGitOps
- Red Hat has a doc on this! https://github.com/gnunn-gitops/standards

Chris Short:
- AWS is working on this, and has NON-PUBLIC docs they set us via Weaveworks/AWS NDA (scott has a zip file)

Roberth Strand:
- adamently agrees we should start with plain yaml, not use kustomize to start (it's only good for folks who are alredy very familiar with kubernetes)
- perhaps start with "repo structure can be as simple as one namespace per app with its manifests and dependencies"
- scott: how did you handle dependencies?
- roberth: we just use flux directly, mostly we don't need to do anythig special for dependencies, but when needed we use Flux's dependsOn
- "we work in an industry of 'it depends'"
- it may not be one size fits all across teams in an org, or even within a specific team
- can mix & match the above strategies - all may be active at the same time
- monorepo thoughts
- depends on your team
- team/namespace make sense for some teams
- another way to use namespaces is one ns per application
- just plan YAML manifests appropriate to each app within that app's namespace
- platform team may have a repo
- no deep dive into the git splitting and merging with git history
- PS, crayon is giong to do gitops, and on a really large scale. Roberth is now head of platform services. All gonna be flux. starting on azure, but will be diong across multiple cloud priders
- Roberth would love to contribute to Terraform Controller, and likely will (tell Chanwit). He would like to contribute to something, rather than creating something new

Thomas Schuetz:
- like rule of thumb approach, and not standardized
- structure reflect a bit of the organiztion, and also a bit of how teams work with each other (example: how app developers work w platform engineers, etc). So when you have one monorepo where that's shared, you can conclude they work together well
- dev, staging, and production. But what if you have multiple production environments?
- us: YES. These are example cluster names not a recommended convention
- this is currently a "stage-centric" appraoch. At keptn there is a service-centric approach (service then stage below)
- this is good for portability - easy to jsut take this folder and move it. We should mention this option at the start under the simple example

Leo:
- is it a good rule of thumb to shoot for consistency of repo structure across an org?
- if you want each team to have autonomy, do you want them to be compeltely decoupled yet properly interfaced so that they can each build at their own pace/style so they can remain interoperable across the org's full solution? If so, to recommend a repo structure per team is breaking this rule. Alternatively, you would want to hook whatever the team's structure is into the overall solution.
- pros of monorepo:
- you can see everything in one place
- cons of repo-per-X (env, team, app, etc):
- no reusbility of code across X
- no single/simple list of all the sources
- if you start by using plain YAML, it will give you a perspective of what exactly you need to deduplicate
- similar to learning to code. Start simple. As you iterate, you identify what can be turned generic/abstracted etc. If you think about configuration the same way, you will have similar levels of learning - rather than trying to determine your highest levels of abstraction up front, because you might not know
- when we say "single source of truth", people often ask "does that mean a single repo?"
- Leo: Not necessarily, but it means "doesn't need to be one repo, but the entire desired state of your ttarget system can be traced to a common origin." if there are multiple repos, perhaps there should be a single entrypoint that lets you track down potential recusion across multiple repositories. Gives visibility/traceability. This is an interesting idea
- Scott: would hesitate to suggest this, because it can give the false illusion that this guarantees covering everything. A monorepo can't GUARANTEE there are no other sources (e.g., you can have a monorepo AND a repo for a single team, AND one-offs for apps)
- OR 👍 "your repo structure defines the scope of your 'truth'"

Hidde:
- brain works by mix and match
- most people don't work in patterns, but in abstract
- often surprised that folks don't know how to use kubectl describe to let the object API itself to describe why it's not working (they instead look in logs). Secondly kubectl has an explain command, will look up embedded documnentation on the API cluster, and will at least tell you how the cluster
- thinks it comes from a shift in people learn these days. He comes from a time where there were no massive amounts of video tutorials telling you step by step how to do things. The newer generation of engineers are used to things being offered ina ready to consume way - which makes you skip the step where you try to get to the core of the problem, understand the core of the problem by looking at in depth resources about the core of the problem
- idea for bridge:
- start with a source (can be used by kustomize, helm, terraform, etc)
- explain how this may be structured within an monorepo to allow composition
- then explain what boundaries are with an organization, and how to mitigate these
- it's basically a jigzaw, where you design different puzzels for different kinds of people