# Best practices for Git repositories consisting of very large number of TEI XML files
This document: https://hackmd.io/@UPPMAX/git-large-tei-xml

## Possible workflow on Bianca
- clone / partial clone to scratch (node-local I/O operations faster)
- commit locally
- push to central repo
- commit less often

## Challenges with Git version control for TEI XML files
- inconsistent indentation, line wrapping, attribute reordering
- ensure validation and schema compliance durring commits and pull requests
- e.g., use tools like `xmllint` and integrate it with pre-commit hooks

## Git version control for repositories with a very large number of files
Ensure you have a recent version of Git as some of the options are new.
- organizes files into subdirectories
- consider:
- submodules
- subtrees
- `sparse-checkout` and partial clone (per-user narrow clones)
- or a combination of the above
- add in `.gitignore` everything that does not need version control
- Configure Git: **CAUTION**: these options need to be tested to see if it improves the performance for this particular repo. Make sure you understand what the config does before implementing the change:
- `git config feature.manyFiles true` lets Git know it's dealing with a large repo
- `git update-index --index-version 4` performs a simple pathname compression that reduces index size by 30%-50% on large repositories
- performance tweaks:
- `git config --global core.preloadIndex true` preloads index in memory, speeds up git status and git diff
- `git config --global core.deltaBaseCacheLimit 512m` affects delta compression performance
- `git config --global core.fsync none` reduces I/O latency, safe for scratch/non-critical repos, would be suitable for local work before pushing to the central repo
- `git config --global core.packedGitLimit 512m` the larger, the better, needs to fit in available RAM
- `git config --global core.packedGitWindowSize 64m` useful during compression/cloning/fetching, value may need tuning, large window sizes increases memory usage
- `git config --global core.untrackedCache true` useful for repos with many untracked/generated files
- git status speed-up and and index optimization:
- `git config status.showUntrackedFiles no`
- `git config diff.renameLimit 0`
- ~~`git config core.fsmonitor true`~~ speeds up git status; update: does not seem to work, unsure about the suitable argument for fsmonitor
- disable git garbage collection (`git config gc.auto 0`), reason: auto garbage collection is too heavy in a shared environment
- maybe???
- `git config pack.threads 1` sets the pack threads to 1

Example of `.gitconfig` (does not include all options above):

[core]
    preloadIndex = true
    deltaBaseCacheLimit = 512m
    packedGitLimit = 512m
    packedGitWindowSize = 64m
    fsync = none
    ignoreStat = true
    untrackedCache = true

[status]
    showUntrackedFiles = no

[gc]
    auto = 0

[pack]
    threads = 1

References:
- https://www.git-tower.com/blog/git-performance/