## What Metrics are?

> **You can’t improve what you don’t measure.** - Peter Drucker

- **Quantitative**: numerical values that can be measured
- **Measurable**: based on data that can be collected
- **Relevant**: should reflect the aspects of success
- **Actionable**: be clear, concise, and easy to understand
- **Timely**: should be updated on regular basis
- **Consistent**: should be reliable and comparable between different scenario
- **Benchmarked**: industry standard or best practice, providing widen context

A metric is not an analysis; rather, it is a **simple combination** of data that indicates a specific fact related to the subject.

## Define metrics of a product

### First, a product is?

A product is something that takes user **inputs**, performs some **actions** on it, and produces **outputs** that fits the user’s **goal**, the metrics are aiming to track those actions performed.

> $y=f(x) \to y' \approx f(x')$

It’s also important to have metrics defined before developing a product for:

- Metrics give a common goal to the team and help maintain focus.
- Metrics hold the team accountable to the users and business.
- Metrics enable more objective decision-making.

### The GAME Framework

- Articulate the **Goals**

The first step to be top-down, which is critical for developing metrics.

A bottom-up approach relies on intuition and is difficult in general.

- User Goals
How will the users benefit from the product?
- Business Goals
What are the tactical or strategic business benefits?

- List the **Actions** that Matter

Should start as a **qualitative** list, and be **comprehensive** but not exhaustive.

There are some frameworks commonly used to flush out the actions.

[What is the AARM Framework and how is used by Business Analysts and Product Managers?](https://www.modernanalyst.com/Careers/InterviewQuestions/tabid/128/ID/5350/What-is-the-AARM-Framework-and-how-is-used-by-Business-Analysts-and-Product-Managers.aspx) 

[Heart Framework: User-Centric Metrics To Measure User Experience](https://aelaschool.com/en/userexperience/heart-framework-user-centric-metrics-measure-user-experience/)

[AARRR Framework Guide: A Step-by-Step Overview of Channels, Tactics and Metrics](https://tracker.my.com/blog/227/aarrr-framework-guide-a-step-by-step-overview-of-channels-tactics-and-metrics?lang=en) 


Overall, the frameworks ask critical questions about how to align the business goals with the user goals.

- Define the **Metrics**

The next step is to turn the desired actions into measurable and trackable value, in other words, turn the **qualitative** indices into **quantitative** metrics.

There are some major decisions for how to count each actions:

- Direct or Proxy
- Individual or Aggregated
- Magnitude or Ratio
- Intrinsic or Heuristic

- **Evaluate** the Metrics

Metrics must be evaluated by tests and iterates, by the means of  applying statistical analysis among the **goals**, **actions** and **metrics.**

Furthermore, the goals may shift over time due to the stage of the product.

## After the Metrics made

### Unintended consequences

Focusing on the only few metrics would risk the business on unintended consequences, caused by the side effects produced by persuading the metrics.duct

### Design Direct Incentives

Simply define high-cost punishments or high-value incentives around the unintended consequences, however, by this nature, this may just overlay on top of the core issues.

### Establish Counter-Metrics Guardrails

By anticipating trade-offs a single metric may create in the extreme scenario, and introduce another metric that measures the other side of the story.

## Metrics Framework

### Metrics Hierarchy

- North Star - Few single truths fit business goal
- Primary - Major attributes that aligns to North Star
- Proxy - Minor attributes that related to Primary metrics
- Guardrail - Identify the unintended consequences

## How Metrics collected

<iframe frameborder="0" style="width:100%;height:543px;" src="https://viewer.diagrams.net/?highlight=0000ff&edit=_blank&nav=1&title=Logging%20%26%20metrics.drawio#R7VxZd9o4FP41nNN5IMcLeHkMa7M2DU1JnuYILIwTxyKyCJBfP3KQ8SLVYfFCMs1Da8nSxdJ3d127prafl30MZtMrZEG3pkjWsqZ2aooia3qD%2Fhf0rNY9hmKsO2zsWGxQ1DFw3iDrlFjv3LGgnxhIEHKJM0t2jpHnwTFJ9AGM0SI5bILc5K%2FOgA25jsEYuHzv0LHIlPXKmhnd%2BA4de0rC9enrG88gHMxW4k%2BBhRaxLrVbU9sYIbK%2Bel62oRtsXrgvw7PV0L180vrnP%2F0XcNe6%2BHX9u74m1ttlymYJGHpkb9Kzp3P15%2Fnt5Ic6f%2Fvl%2F9vpXN7pdYblK3DnbL%2FYWskq3ECM5p4FAyJSTW0tpg6BgxkYB3cXlGVo35Q8u7Ql08uJ47pt5CL8Ple1mtCwGqx%2FwGiCOUG0C1EiDgm4SgkI%2BwSjJxibaygjVdPonS1Xz3bpFWIClzHs2W70IXqGBK%2FoEHa3aZ6YjLkZbzcMBvUiximSIp1I2rp%2FGmMUWWKDAeNQe%2FMLEQr0ggEhBuV8MVzdNRe3Vw%2FSy0v71VGuv79sVlIQKJMJ1Mbj%2FUCxdHMkSaWCojaOAhSpYEkB0JjsCYo2NuBoUiAompQUEzEgAjSUHNAg7cfOm%2FT60DKGP%2FuLm3F%2FOTuvN7nNhxZV%2B6yJMJkiG3nA7Ua9rSQ80ZhLhGYMlEdIyIrZMLb5McjoruLVfTD%2FpBk2Hxi590ZnmWitwtbSIbFptPUQUqTX0aSgEc7ZDUofzfEYZo0LTS%2FANmQEf5zddgd3sxvS7z%2FNb8dg0SercGOD3czkDAxdQJzXpJEVwfw%2B9RRjsIoNmCHHIz7HBRv6%2BzOGzonpwPGehNxyCUbU10kgDFzH9uj1mO46pNLVCmTEoc7EKbvx7FjWmpmg77yB0Tu9AC%2B2Ikq82ao1O0IEsziZuT%2BMYuR0bC%2Bk0olsJhXnurUbUs%2FLm2AlEdm6miBabyQJoMnEpwxVAJIah%2BQlsusj4FMRVjSX7mlrhOmVHVxd0R9zxn4lWoFJtxyT7UjSxdIdaZKEHonUyh80SQlaQTxQqUgtROyYK8Nl7kaM49oumlNukyjn2Y5nU24Dz4Ed90b%2BbAOH0AOQt%2FAAkEcYV9H9VVu2C3yfAb228WHAEozeRB9Fel%2BqlLT0ssghFln6RlGGnsfEJxbEuFr7X6nUqtsKrZy30G6LZNZjx5DsX3R5Tf6tA%2BjKxlM6tAXGT9Cz%2FvlyYmZyQU71grZFJuCvMc00psqWLrZ6VGIZPrbA6l0hzyEI%2Fx8MX1OSTqT4n3Zk4ml%2BFmnkshO9nkr%2FDgif95fTLPH7UEwbhbi8XHwlS3oqu2KmeGj9pGxaxEZ%2Fdp7FP2SmsjiKJKW4ck0x1yBObnDaZROqSR3gT0cIYEvI2wWH54ESEnEpz21Z8phH2C4FP5xAppZL4K4IqeYaR2UJWNzRw2ACPPDlrIZ%2BdOGSUnFidDcPTGxf9H0NTM6GorGlodAKMRS7qndZTjGjYqbOAz%2Ba0NSyJxiHDVdKMDfh2e0xuElKMVELx%2F5rf7UU9heb%2BLKPDLblf31H%2Fjd24ufdRss5c38WYjHDO4Qj2nF39uUsrylF578h3lrVxjdMuB2D9jm2lImw%2BkTaVsVIeauYbXEWPnY1MJcOWdb5cByyLGiPBLEmpxm7vy6FKJZ%2BKJzFYXlEl0o6lxUSPzS8DBOHKbLFx5c8mC3Hph0vcxg8MHegQG%2FdggX99%2BsdJKjZacvqTaL6mVRlRupyU8Gjx0t4NnY15xKerDDz6Cp4xOpBSXvjRgnhH%2B8Apw8uussZZagqNP%2BB%2BUY1L4tQpxpDl40EOAdahNACpKv9cjEB9xf1XnPY6f8weo831uDOnhg%2FQu%2FiMznayTMPeW9VVLxWKckF30ut6FqxWSWhN8YntIMaBdozWO8u53Ict37Jkqhc9Issm8lKxPqBJxopMsW7mHzt4XsmpXeFRo4rAPxbD9Pt%2F5K1KrKckjipap8yDDw%2Bk%2FYvK2bP1N3lH3kfhjOvd29hVe7blm5ayJv5%2BGnNpBY9sKA7Rab4Cu6Q7xIl3FSH4cB0krnloC%2BnLdMnwtVH4LL2WbTln0%2B2jkWPCpKf4oF6Ic4yH2QLCknV7aqHcpNxPhsnLusdQPzqBM5y9WW9eR2hHlTwW70bZXx%2BxZC3gOu8gJdUWr9XNNxIRcOpU%2BaPx39QwqFmjy8m2lY4hTLGyA%2FqBWcYPQafCKBIHvc7faFo5eIFKqaqJ1E4zA0MKTflBFWtNK%2BQf0WTe83qf2MYuBeuyjQLQ%2Bvt6vXs7P6yfv5wQZpweG9%2FF7znfj4fofoN5p11ukZSnKwFtsNnOHLYeMgLqMRfog%2B7YvDrYZs987Yf0tjdvhvJUE34hrwAxbSzltuBd6MS275nsnt%2FK27ddHtPq%2BuFvjQH9svi8fepdSGqKjmuEoWsp%2BaKt05vzgTe9KltY2gDErwLLQXZ6AO96aIEo5nSb2qZFdLC73jw7sW7frs%2B%2FaveMsKXFIoC%2FWaWqd54YfmwBKQdZJ4Cafl6KfoPXl4rNQclhMvk4OojZAcHKdIpNTArIvp0w1%2Fx24ifkhQ%2F4WeqtHzkjzajL5qtXf7ou3Bq9z8%3D"></iframe>

## What to do next

- Prove improvement by data
- Evaluate contribution finely
- A/B testing

## References

[4 Steps to Defining GREAT Metrics for ANY Product](https://hackernoon.com/metrics-game-framework-5e3dce1be8ac)

[The ABCs of Defining Product Metrics for your Team](https://www.howtoweb.co/the-abcs-of-defining-product-metrics/)

[How to design a winning metrics framework](https://www.airops.com/blog/how-to-design-a-winning-metrics-framework)

[How to Choose & Measure North Star Metrics: Acquisition, Retention, & Monetization](https://www.reforge.com/previews/growth/north-star-metrics)

[What is a “North Star Metric”?](https://growwithward.com/north-star-metric/)