owned this note
owned this note
Published
Linked with GitHub
###### tags: `sbc`, `calibration`
# SBC idea genration: prior
: priordb to sync prior knowledge on SBC
please help us generate reliable posterior which we can fit together during session!
how? report missing literatures
Theoretical support
* [Validating Bayesian Inference
Algorithms with Simulation-Based
Calibration](https://arxiv.org/pdf/1804.06788.pdf) Talts, Betancourt, Simpson, Vehtari, Gelman, 2018
* [Rank-Normalization, Folding, and Localization: An Improved R-hat for Assessing Convergence of MCMC](https://arxiv.org/abs/1903.08008) Vehtari, Gelman, Simpson, Carpenter, Bürkner, 2021
* [Graphical Test for Discrete Uniformity and its Applications in Goodness of Fit Evaluation and Multiple Sample Comparison](https://arxiv.org/abs/2103.10522) Säilynoja, Bürkner, Vehtari, 2021
* [Toward a principled Bayesian workflow in cognitive science](https://psycnet.apa.org/record/2020-43606-001) Schad, Betancourt, Vasishth
* [Bayes factor workflow](https://arxiv.org/pdf/2103.08744.pdf) Schad, Nicenboim, Bürkner, Betancourt, Vasishth, 2021
* [ECDF with codes](https://avehtari.github.io/rhat_ess/rhat_ess.html)
Application support
* [Cognitive science, response time fitting](https://link.springer.com/content/pdf/10.3758/s13428-019-01318-x.pdf)
* [Bioinformatics, effect of mutation prediction](https://www.biorxiv.org/content/10.1101/2020.10.27.356758v1.full.pdf)
* [Earth science, earthquake prediction](https://gmd.copernicus.org/articles/11/4383/2018/gmd-11-4383-2018.pdf )
Tutorials
-
# SBC idea genration: likelihood
: how the audience wish to use SBC
please help us generate reliable posterior which we can fit together during session!
how? report description format with 1. target validation (model vs algorithm) and parameter, 2. domain/purpose/data, 3. code or further model description if possible.
eg1.
1. target: model
- hierachical spline
- $\sigma_{\bar{\alpha}}$: is prior given to parameter $\sigma_{\bar{\alpha}}$ which represent the degree of homogenioty between different types of engine.
2. background
defense industry/predict engine failure function/683 data points (annual failure count) from 99 ships
$\begin{aligned} Y_{s} & \sim \operatorname{Normal}\left(\mu_{s}, \sigma_{y}\right) \\ \mu_{s} &=\alpha_{s}+\Sigma_{k=1}^{K} w_{k, s} B_{k} \\ \alpha_{s} & \sim \operatorname{Normal}\left(\overline{\alpha_{e}}, \sigma_{\alpha}\right) \\ w_{s} & \sim \operatorname{Normal}\left(\overline{w_{e}}, \sigma_{w}\right) \\ \overline{\alpha_{e}} & \sim \operatorname{Normal}\left(\overline{\alpha_{0}}, \sigma_{\bar{\alpha}}\right) \\ \overline{w_{e}} & \sim \operatorname{Normal}\left(\overline{w_{0}}, \sigma_{\bar{w}}\right) \\ \sigma_{y} & \sim \text { Exponential }(1) \\ \sigma_{\alpha} & \sim \operatorname{Gamma}(10,10) \\ \sigma_{w} & \sim \operatorname{Gamma}(10,10) \\ \sigma_{\bar{\alpha}} & \sim \text { Exponential }(1) \\ \sigma_{\bar{w}} & \sim \text { Exponential }(1) \end{aligned}$
3. [sbc code](https://github.com/hyunjimoon/defense-reliability/blob/master/spline/sbc_hierarchical_navy.R) and [model description](https://arxiv.org/abs/2012.01224)
eg2. -> need further details from Charles
1. target: algorithm
- embedded laplace
- $\lambda, p$ does the embedded laplace algorithm show bias for Bernoulli-Logistic Latent Gaussian Models?
2. background
Latent Gaussian Models/prostate cancer data
# SBC idea genration: algorithm
: how the audience could implement SBC
please help us generate reliable posterior which we can fit together during session!
how? report relevant ideas with regarding but not limited to the following.
- tutorial during session
- SBC connection with posteriordb [issue](https://github.com/stan-dev/posteriordb/issues/111) and brms [issue](https://github.com/paul-buerkner/brms/issues/732)
- SBC interface: Mike Lawrence made SBC shiny dashboard [here](https://github.com/mike-lawrence/sbc_demo)
- SBC computation solution (need feedback from):
-- parallelize sampling or fitting (Steve)
-- bootstrap data or parameter
-- early stopping ideas
-- rejection sampling eg. fold-change $\in$[0,1] in [this](https://www.pnas.org/content/116/37/18275) paper (Martin)
- SBC diagnose measures: metric (goodness of fit, IPM, f-divergence) and graph (ecdf, decdf)