---
# System prepended metadata

title: Performance Evalutaion Methodology
tags: [Profiling]

---

###### tags: `Profiling`

# Performance Evalutaion Methodology

:::info
Linux system performance by Brendan Gregg

http://www.brendangregg.com/Slides/LISA2019_Linux_Systems_Performance.pdf

![](https://i.imgur.com/e15bBoG.png)

:::

## Overview

Common questions in performance evalutaion

1. What is the function consuming most cycles? and its context (who are calling it?)
2. What is the application consuming most cycles
3. What library is most used?
4. Do we have a lot of TLB page walks? On which functions, binaries?

### Intel SkylakeX core PMU

PEBS: Precise sampling to eliminate interrupt-based sampling IP-skid
- Sample recorded by microcode to virtal memory buffer
- 1 interrupt per buffer full: **significant overhead reduction** = increased sampling frequency
- Capture IP, machine state, data address for ld/st

## Top-Down Analysis

Look at the events below listed by perf

```bash
  topdown-fetch-bubbles OR cpu/topdown-fetch-bubbles/ [Kernel PMU event]
  topdown-recovery-bubbles OR cpu/topdown-recovery-bubbles/ [Kernel PMU event]
  topdown-slots-issued OR cpu/topdown-slots-issued/  [Kernel PMU event]
  topdown-slots-retired OR cpu/topdown-slots-retired/ [Kernel PMU event]
  topdown-total-slots OR cpu/topdown-total-slots/    [Kernel PMU event]
```

Perf adds these events to enable user to do top down microarchitecture performance analysis[^1]. Proposed by Ahmad Yasin at intel, the method allows us to investigate how a workload or an application is utilizing hardware resources. This can be done at a very low cost since it uses on-chip PMU directly.

[^1]: A Top-Down method for performance analysis and counters architecture

## Tracing Distrubuted Systems

- machine-centric: Difficult to show the relations and dependencies between multiple nodes and servers.
- workflow-centric: also known as end-to-end tracing. For example, a request-based distributed service, a trace records the work done in servers.
    - Notable works: Stardust, X-Trace
    - Google’s Dapper, Cloudera’s HTrace, Twitter’s Zipkin

End-to-end tracing capability:
- anomaly detection
- diagnosis of steady-state correctness and performance problems
- profiling (done by sampling trace)
- resource-usage attribution

### Approach to End-to-End Tracing

- Metadata propagation
- Schema-based
- Black-box inference

## References
- Hardware Performance Monitoring Landscape:　https://protools19.github.io/slides/Eranian_KeynoteSC19.pdf
- https://zhuanlan.zhihu.com/p/60940902
- https://software.intel.com/content/www/us/en/develop/documentation/vtune-cookbook/top/methodologies/top-down-microarchitecture-analysis-method.html