intro

key points

• we incentivize early prediction accuracy by giving more weight to such submissions.
• we introduce a differentiation mechanism to reward more top miners along the obvious fact that (in terms of Brier score) an accuracy of 0.80 is much better than 0.76 - this mechanism is currently linear. It will be made exponential soon after to also reflect the also obvious fact that improving from 0.89 to 0.90 is exponentially harder than improving from 0.75 to 0.76.

definitions

Let K be the amount of miners, E an event,

$I$ the first time

$E$ is listed on our network, D the cutoff, and

$T$ a threshold.
Let

$S(p,o_E)$ be the Brier score of a prediction

$p$ sent for the event

$E$ which resolution was

$o_E$ (1 if it occured and 0 otherwise).
Let

$(p_{t,k}{)}_{t\le D}$ be the time series of all the predictions submitted by miner

$k$ for

$E$ before cutoff.

The scoring rule

We want each prediction to have a corresponding weight depending on the time of the submission

$t$. Let then

$w_{t,D}$ be the weight of the prediction

$p_{t,k}$. It is a function of the submission date

$t$ and the cutoff date

$D$.

When the event

$E$ resolves we compute the Brier score of each submission and we multiply this score by the corresponding weight. We then obtain the following weighted time series:

$(w_{t,D}\times S(p_{t,k},o_E){)}_{t\le D}$

The final scoring rule is obtained by averaging this quantity. We detail all the steps below as well as add an additional discretization step.

weights

We choose exponentially decreasing weights along the intuition that predicting gets exponentially harder as one goes back in time.

We divide the time segment

$[I,D]$ into

$n$ intervals

$[t_j,t_{j-1}]$ of equal length (currently 4 hours). Then for the interval

$[t_j,t_{j-1}]$ we set the weight

$w_{j,D}=e^{-\frac{n}{j}+1}$ where

$t_n=I$ and

$t_0=D$ and where

$j$ decreases from

$n$ to

$1$.

averaging per window

Each

$p_j$ is the arithmetic average of the miner's predictions in a given time window

$[t_j,t_{j+1}]$ i.e

$p_j=\frac{\sum _{t^{\prime }\in [t_j,t_{j+1}]}{p}_{t^{\prime }}}{\sum _{t^{\prime }\in [t_j,t_{j+1}]}1}$

main scoring component

Given our time series

$(w_{j,D}\times S(p_{j,k},o_E){)}_{j\le n}$
we compute the time weighted average

$m_k$:

$m_k=\frac{\sum _j{w}_{j,D}S(p_{j,k},o_E)}{\sum _j{w}_{j,D}}$

the full scoring rule

We add an exponential to obtain the final scoring rule with parameters

$\alpha$:

$$\exp (\alpha \times m_{k,E})$$

We have

$\alpha$ set currently close to

$25$ so that top miners close to

$0.9$ average Brier score would be able to get 10% of the reward. We expect to increase

$\alpha$ moving forward.

If one then denotes:

$L_k=\exp (\alpha \ast m_{k,E})$

We normalise the scores across all miners using the

$l1$ norm:

$s_k=\frac{L_k}{\sum L_j}$

We increment a daily average

$s_{ki}$ denotes the above score in the case of event

$i$, we have

$S_{daily}=\frac{1}{n}\sum _k{s}_{ki}$

This resets every 24 hours.

We continuously compute an EMA

$\alpha S_{daily}+(1-\alpha )S_{prev}$