---
# System prepended metadata

title: Security Metrics

---

# Security Metrics
## What to measure?
![](https://i.imgur.com/6XhtycR.png)

### Security cost
- Direct costs
    - Purchasing, installing and administering security measures
    - Hiring security staff
- Indirect cost
    - Employee's time to recover from a data breach
    - Business disruption due to software patching
    - Retraining employees
- Fixed cost: independent of the activity in the core business
    - Acquiring IDS, firewall
- Variable cost: grow proportionaly to the activity
    - Distributing security tokens to customers
- Onetime costs
    - Acquisition and deployment of protection measures
- Recurring costs
    - Hardware maintenance, software updates, etc.
### Security level
- Deterministic indicators
    - User awareness
    - Virus scanners
    - Software vulnerabilities
- Stachastic indicators
    - Compromised machines
    - Phished credentials
### Security benefit
- reduction of losses incurred in the absence of security
- Benefits depend on the value of the assets at risk
## Measuring security levels
- Security level is a '*latent construct*'
    - meaning it can't be observed or measured directly
- We can only measure indicators or metrics that reflect differrent aspects of the latent construct
### Controls
- the measures you put in place to mitigate risk
    - Physical: door locks
    - Organizational: incident response team
    - Procedural: credentials policy
    - Techniacl: encryption, firewalls
### Vulnerabilities
- Weaknesses in the controls that could be leveraged by a certian type of attack
- Finding *known* vulnerabilities: vulnerability scanners, CERT alerts, etc.
- Finding *unknown* vulnerabilities: penetration testing, red teaming, ,etc.
### Incidents
- Events where security is compromised in some form
- Some succesful attacks could go undetected for months or even years
### Prevented losses
- Mapping incidents to losses is hard
- Mapping *prevented incidents* to *prevented losses* is even harder
- Did incidents decrease because of failed attacks or fewer attacks?
![](https://i.imgur.com/I8SOOCZ.png)

## Metrics in practice
- Applying the framework to concrete examples
    - Cloud Security Alliance Cloud Control Matrix
    - Security Service Level Agreement (SLA)
    - Software Security Maturity Model
    - Cyber Security Assessment Netherlands
### CSA Controls Matrix
- Set of 133 controls identified by Cloud Security Alliance, used to assess and rank security of cloud services
- All control-based metrics (no surprise there!)
- Vulnerabilities are mentioned, but only in terms of the controls needed to deal with them
![](https://i.imgur.com/F4rqLmh.png)

### Service Level Agreement (SLA) Security Metrics
- "Hosted Email Security SLA" (Trend Micro)
    - Guarentee availability of email service by providing lots servers, not by ensuring the level of security
    - ![](https://i.imgur.com/5m8HySb.png)

### Building Security in Maturity Model (BSIMM)
- Observational model built from real-world software security initiatives
- Metrics based on 12 practices with 110 associated activities
- Activities include:
    - Penetration testing
    - Provide awareness training
    - Code review
    - Identify software defects found in operations monitoring
![](https://i.imgur.com/6vxR6av.png)
![](https://i.imgur.com/7hkyVE2.png)

### Cyber Security Assessment Netherlands
- Rates security threats qualitatively
- Forward looking, evaluates controls in light of emerging vulnerabilities and incidents
![](https://i.imgur.com/ZhA4OmI.png)

### Conclusion
- Many metrics focus on controls
- Especially when they have been developed by security industry or sevice providers
- Why?
    - Easier to measure
    - Selling controls is the business model
    - Controls are about effort, not about results against attackers
    - Focusing on controls leaves the risk of failure with the buyer
## Metrics from incident data
- Many available metrics are based on controls and vulnerabilities
- What is the effect of this bias?
- Many controls (standards, certification, auditing) are assumed to correlate with higher security, but in reality we often don't know nor have evidence to the contrary
> We can't trust controls to do what we think they do. We have to study the actual effects
- Leaving out the interaction with the threat environment means we can't tell how security policies perform in the real world
- Incidents provide important feed back regarding where to allocate your resources
- Incident data are driven by stochastic metrics (e.g., attacker behaviors)
- Incidents only reflect on what has already happened, and don't necessarily reflect upcoming threats