# Parameter Selection Under Uncertainty (PSuU) Information Repo

## Summary

PSuU is a workflow for standarizing experiments and analysis for enabling agile scientific iterations over recommendations on multi-stakeholder scenarios that are based on complex models. The process can be summarized as follows:

1. System Goals are identified
2. Numerical KPIs that serves as proxies for each goal are identified
- Each KPI aggregates the time dimension into a single number
3. Success criteria for each KPI in terms of the System Goals are identified
- For example, the sucess on a system goal can make use of normalized mean of the KPIs, or specific thresholds on certain KPIs.
- Another way of putting it, is that **this is a map between a set of KPIs to a single number between 0 and 1**
5. Search Regions over parametrizations are identified.
6. Parameter are separated between **Control Parameters** and **Environmental Parameters**
7. Simulations are executed and results dataset are generated
8. KPIs for each run are calculated, and success on system goals are aggregated.
9. A **feature dataset is constructed** by using the simulation **Control Parameters as features**, and the **system goals as labels**. This allows for using Machine Learning techniques in order to perform a variety of tasks, like dimensionality reduction, sensitivity analysis, non-linear fitting on the goals, among others.

The analytic goal is to:
- Make explicit what are the ranges of parameters that maximizes the success of the design for a set of system goals that can be compete against each other. Normally, the optimization criteria is against a "combined system goal" which is a weighted average of the goals success metrics.
- Make explicit what are the trade-offs of optimizing over different system goal weightings. Specifically, it is important to know if the optimal design is robust in terms of optimizing against each goal individually, or against different combinations of goal weights.

## Pointers

- [How to Perform Parameter Selection Under Uncertainty](https://medium.com/block-science/how-to-perform-parameter-selection-under-uncertainty-976931ba7e5d)
- [cadCAD Machine Search library](https://github.com/cadCAD-org/cadCAD_machine_search): Library encapsulating the PSuU primitives together with some common visualizations. Under development.
- [Bosch Workshop #4: Understanding the cadCAD analysis workflow](https://drive.google.com/drive/u/1/folders/1rBriVpzXLU_15QcoyA2lUbImFNWxhDP5)
- [Interacting AMM Model](https://github.com/cadCAD-org/interacting-amm-model): An public implemented version of a cadCAD simulation going through the entire PSuU process
- KPIs and System goals: https://github.com/cadCAD-org/interacting-amm-model/blob/master/interacting_amm_model/kpis.py
- Analysis: https://github.com/cadCAD-org/interacting-amm-model/blob/master/notebooks/analysis-single.ipynb
- [Filecoin Documentation](https://drive.google.com/file/d/1twiadZwl5AY9O4EUWc1Dl4wRhDT_nmT5/view?usp=sharing). Report that uses the PSuU for providing quantitative recommendations
- [Old cadCAD model for Filecoin](https://github.com/blockScience/filecoin): First internal implementation of the PSuU methodology.