# Security Metrics
## What to measure?

### Security cost
- Direct costs
- Purchasing, installing and administering security measures
- Hiring security staff
- Indirect cost
- Employee's time to recover from a data breach
- Business disruption due to software patching
- Retraining employees
- Fixed cost: independent of the activity in the core business
- Acquiring IDS, firewall
- Variable cost: grow proportionaly to the activity
- Distributing security tokens to customers
- Onetime costs
- Acquisition and deployment of protection measures
- Recurring costs
- Hardware maintenance, software updates, etc.
### Security level
- Deterministic indicators
- User awareness
- Virus scanners
- Software vulnerabilities
- Stachastic indicators
- Compromised machines
- Phished credentials
### Security benefit
- reduction of losses incurred in the absence of security
- Benefits depend on the value of the assets at risk
## Measuring security levels
- Security level is a '*latent construct*'
- meaning it can't be observed or measured directly
- We can only measure indicators or metrics that reflect differrent aspects of the latent construct
### Controls
- the measures you put in place to mitigate risk
- Physical: door locks
- Organizational: incident response team
- Procedural: credentials policy
- Techniacl: encryption, firewalls
### Vulnerabilities
- Weaknesses in the controls that could be leveraged by a certian type of attack
- Finding *known* vulnerabilities: vulnerability scanners, CERT alerts, etc.
- Finding *unknown* vulnerabilities: penetration testing, red teaming, ,etc.
### Incidents
- Events where security is compromised in some form
- Some succesful attacks could go undetected for months or even years
### Prevented losses
- Mapping incidents to losses is hard
- Mapping *prevented incidents* to *prevented losses* is even harder
- Did incidents decrease because of failed attacks or fewer attacks?

## Metrics in practice
- Applying the framework to concrete examples
- Cloud Security Alliance Cloud Control Matrix
- Security Service Level Agreement (SLA)
- Software Security Maturity Model
- Cyber Security Assessment Netherlands
### CSA Controls Matrix
- Set of 133 controls identified by Cloud Security Alliance, used to assess and rank security of cloud services
- All control-based metrics (no surprise there!)
- Vulnerabilities are mentioned, but only in terms of the controls needed to deal with them

### Service Level Agreement (SLA) Security Metrics
- "Hosted Email Security SLA" (Trend Micro)
- Guarentee availability of email service by providing lots servers, not by ensuring the level of security
- 

### Building Security in Maturity Model (BSIMM)
- Observational model built from real-world software security initiatives
- Metrics based on 12 practices with 110 associated activities
- Activities include:
- Penetration testing
- Provide awareness training
- Code review
- Identify software defects found in operations monitoring

### Cyber Security Assessment Netherlands
- Rates security threats qualitatively
- Forward looking, evaluates controls in light of emerging vulnerabilities and incidents

### Conclusion
- Many metrics focus on controls
- Especially when they have been developed by security industry or sevice providers
- Why?
- Easier to measure
- Selling controls is the business model
- Controls are about effort, not about results against attackers
- Focusing on controls leaves the risk of failure with the buyer
## Metrics from incident data
- Many available metrics are based on controls and vulnerabilities
- What is the effect of this bias?
- Many controls (standards, certification, auditing) are assumed to correlate with higher security, but in reality we often don't know nor have evidence to the contrary
> We can't trust controls to do what we think they do. We have to study the actual effects
- Leaving out the interaction with the threat environment means we can't tell how security policies perform in the real world
- Incidents provide important feed back regarding where to allocate your resources
- Incident data are driven by stochastic metrics (e.g., attacker behaviors)
- Incidents only reflect on what has already happened, and don't necessarily reflect upcoming threats