# XP Fault Tolerance

- Ordering and Coordination requirements
- Participants and Coordinators need to be in agreement about what the last global aggregate was
- Push the messy + hard parts to S3 (or similar)
- allow both coordinators and participants to read the most recent aggregate
- facilitates lower statefulness
- If a coordinator ever observes that a participant has a more recent global aggregate
- it indicates potential undesirable situations:
- another coordinator exists
- a race happened between the last aggregate write and the initialization of this particular coordinator
- a misconfiguration has occurred
- the participant is lying
- training sessions run in lockstep
- there are no partial rounds
- but there are minimum thresholds for participation in each round
- Coordinator needs at least 3 participants (GDPR reasons)
- have explicit boundaries (e.g. limit size of the model)
- By setting hard limits, system reliability is easier to reason about
- model size
- 512mb-1g is a likely model size
- compression may be easily enabled over the wire when using grpc
- diffs are possible but may require more coordination
- network latency
- aggregation computation
- Coordinator may need to keep state between rounds
- If we need to restart the coordinator we can continue the training
- Maybe most of the information required may be part of the participant coordinator communication. This would reduce / eliminate the need to persistent information between rounds
- maybe Coordinator has only last global model as state, hash of that global model could act as proof that sent locally updated model used that global model
- The platform needs to store all model versions
- Coordinators can be stateless, and when restarted they can pull the last version to work against
- (implementation detail) put models in S3, let Amazon handle fan-out
- Slow participants may introduce unfairness because their results would never be used
- If we wait for our threshold number of participants and immediately aggregate, only the fastest N will be used
- Handle this in the selection process, change which nodes are requested
- Push vs Pull - hybrid
- Participants can be configured to connect to a XAIN coordinator endpoint
- The coordinator can then request work from the participants that have connected
- This allows coordinators to require low-configuration (don't need to know all dynamic participants)
- This allows coordinators to be connected to, rather than requiring participants to be connectable
- This allows coordinators to avoid overload (participants don't push models until requested)
- Strategies for handling overload
- Overload will probably happen, client workloads can be unpredictable
- Anticipate it, it doesn't need to be stressful
- [USE - utilization, saturation, errors](http://www.brendangregg.com/usemethod.html)
- Utilization Metrics:
- aggregation latency
- bytes read
- bytes written to external storage
- clients connected
- coordinator heap size
- compression ratios
- Saturation Metrics (the extent that resources must be waited for)
- number of participants ignored due to the coordinator not being able to keep up
- number of participants ignored due to an operator-configured participant limit
- number of global aggregates not yet fully uploaded to external storage (S3 or similar)
- number of participants waiting for a coordinator signal to compute a round
- number of participants waiting for a coordinator signal to submit a computed round
- Errors
- connection errors to external storage
- connection errors to the coordinator (requires a separate reporting endpoint for participants to ping, or the use of a "dummy" participant that ensures connectivity is possible from the outside world to a coordinator)
- round failures due to lack of 3+ participants
- coordinators encountering participants who claim to have a more recent global model than what the coordinator has
- Allow operators to reduce the number of participants on-the-fly
- Participants should send heartbeats at a slow pace, and never send a model until requested to do so by the coordinator
- At some scale and at some development stage, it makes sense to shard participants across several coordinators, and then perform a second aggregation on their computed models.
- If coordinators are writing their models to shared storage (S3 or similar)
- The second aggregation can just read the intermediate models from this storage
- Strategies for smooth updates
- Having a generic proxy in front that can be configured to send some clients to a new version
- State minimization
- Model aggregation versus gradient aggregation
- we want to support model aggregation as this does not commit to particular machine learning algorithms
- this maximizes coverage and market share