###### tags: `one-offs` `puzzles` # Unbiased Estimators of the Inverse of the Mean from a Poisson Stream **Overview**: In this post, I describe an estimation puzzle related to Poisson laws, and describe a simple solution. ## The Puzzle Suppose that you have a machine which generates iid $\mathrm{Poisson}(\lambda)$ samples, with $\lambda$ unknown, and I want a non-negative, unbiased estimate of $\lambda^{-1}$. You can take as long as you need, but after a finite amount of time, you have to give me an answer. How do you do this? ## First Thoughts Note that from a single Poisson sample, many quantities admit simple estimators, e.g. - For $n \in \mathbf{N}$, $\mathbf{E} \left[ \mathbf{I} \left[ X = n \right] \right] = \frac{\lambda^n \cdot \exp(-\lambda)}{n!}$. - For $k \in \mathbf{N}$, $\mathbf{E} \left[ X(X-1)(X-2)\cdots(X-k+1) \right] = \lambda^k$. - For $t \in \mathbf{R}$, $\mathbf{E} \left[ \exp (t \cdot X) \right] = \exp(\lambda\cdot(\exp (t) - 1))$. In practice, when taking a fixed number of Poisson samples, one can either take simple averages of these basic estimators, or use tricks related to the Rao-Blackwell theorem to obtain improved estimators. Anyways, let's see whether there is a viable estimator which is computable from just one sample. Suppose that when $X \sim \mathrm{Poisson}(\lambda)$, $f(X)$ is an unbiased estimator of $\lambda^{-1}$, for any $\lambda > 0$. Then, it holds that *as functions of $\lambda$*, \begin{align} \lambda^{-1} &= \sum_{n \geqslant 0} \frac{\lambda^n \cdot \exp(-\lambda)}{n!} \cdot f(n) \\ \lambda^{-1} \cdot \exp(\lambda) &= \sum_{n \geqslant 0} \frac{f(n)}{n!} \cdot \lambda^n. \end{align} This would relate $f$ to the Taylor expansion of $\lambda \mapsto \lambda^{-1} \cdot \exp(\lambda)$ around $\lambda = 0$. But there is a hitch: this function does not admit a Taylor expansion at $0$! As such, this estimator cannot exist. ## Second Pass When a single sample does not suffice, sometimes a fixed number of samples will be enough, i.e. perhaps one could draw $K > 1$ independent Poisson realisations, and combine them somehow. However, one can show that this strategy will also not work here. Roughly speaking, any such strategy could be used to devise a single-sample estimator of $(\lambda \cdot K)^{-1}$ from a single realisation of a $\mathrm{Poisson}(\lambda \cdot K)$, and from our earlier discussion, we know that this is not possible. The next thing to try is then to draw a random number of samples, perhaps adaptively. For example, one could try to design a procedure which is based around drawing independent Poisson variables until one of them returns a nonzero result. This would be equivalent to simulating a Geometric random variable with success parameter $p = 1 - \exp(-\lambda)$, and so the expected number of failed trials satisfies \begin{align} \mathbf{E} [ F ] &= p^{-1} - 1 \\ &= (1 - \exp(-\lambda))^{-1} - 1 \\ &= ( \exp(\lambda) - 1)^{-1}. \end{align} As $\lambda \to 0^+$, this behaves like $\lambda^{-1}$, which suggests that we might be on the right path. I mean this in the following sense: if we are interested in estimating quantities of the form $g(\lambda)$, life is broadly easier if $g$ is smooth, continuous, etc. As such, we might hope to 'absorb the singularity' at $\lambda = 0$ somehow. To this end, write \begin{align} \lambda^{-1} &= ( \exp(\lambda) - 1)^{-1} \cdot \frac{\exp(\lambda) - 1}{\lambda}. \end{align} Bearing this in mind, it becomes clear that if we are also able to form an independent unbiased estimator of $\frac{\exp(\lambda) - 1}{\lambda}$, then we are done. Now, suppose that we implement this procedure, generating $F$ realisations taking the value of $0$, and then generating $X_*$ with the law of $\mathrm{Poisson} (\lambda)$ conditioned to be positive, i.e. with mass function \begin{align} \mathbf{P}_\lambda (X_* = n) := (\exp(\lambda) - 1)^{-1} \cdot \frac{\lambda^n}{n!}. \end{align} As with standard Poisson variables, the expectations under this law of several expressions involving rising and falling factorials are tractable. In particular, one can compute explicitly that \begin{align} \mathbf{E}_{\lambda} \left[ \frac{1}{X_* + 1} \right] &=\lambda^{-1} - (\exp(\lambda) - 1)^{-1}. \end{align} It thus follows that $F + \frac{1}{X_* + 1}$ has expectation $\lambda^{-1}$, and we are done. As an aside (which is not strictly needed for the unbiasedness), it can be checked that $X_*$ and $F$ are independent of one another. ## Closing Comments Some further refinements of this puzzle which might be worth considering are: - Analyse the variance of this estimator, and describe a strategy for controlling the variance (either in absolute or relative terms) uniformly in $\lambda$. - Given $K$ replications of $\{ (F_k, X_{*, k}) \}_{k \in [K]}$, one can improve the estimate of $\lambda^{-1}$ by simple averaging. Can one do anything smarter, in terms of variance reduction? - Similarly, for $K \in \mathbf{N}$, devise an estimator of $\lambda^{-K}$ which makes optimal use of the samples which are generated along the way.