###### tags: `one-offs`

# Lessons from the Bias-Variance Trade-Off

**Overview**: In this note, I discuss how 'exact' solutions to computational problems need not be the be-all and end-all, and - like all computations - are a tool which can be traded off like any other.

## Complexity, Optimality, and Purity

One important psychological aspect of coming to terms with the bias-variance trade-off (or variants on the same idea) is to comfortably let go of the desire for "pure" solutions to problems, and to accept that "optimality" can wear a different face to what one might initially think.

When one is presented with a solution which is { 'exact', unbiased, etc. }, one then develops the reflex to identify parts of the solution which can be sacrificed, in order to improve performance in other ways (e.g. speed, variance, …). An 'exact' solution is not meaningfully perfect, but rather a useful tool which can be traded off in various ways.

Still, trading things off is not always necessary; sometimes the "pure" solution is sufficiently good that the benefits of sacrificing that purity are fairly marginal. In these settings, deviating from the nice solution could just mean that you have one fewer interesting true fact to share about your method, and true facts are nice to have.

For me, this topic is periodically on my mind because some areas of research (e.g. optimisation) are quite explicit about studying procedures which contain inexact sub-components. When you see these presentations, it's natural to think that maybe they should be the norm; addressing inexactness and robustness seems very healthy. This feeling is particularly strong as a statistician, e.g. should we ever honestly make the assumption that our model is well-specified?

A related comment, for those who are familiar with Multi-Level Monte Carlo techniques: in the context of MLMC, one eventually learns that _even if_ you have access to an exact estimator of some quantity, it can be worth your while to find a worse estimator which is cheaper, and then re-allocate your resources. Again, one encounters this story that even paying a good price for something with great properties need not be the optimal decision.