# A Phish and a Detection Engineer Walk Into a Bar…
We are going to talk about how we use Detection-as-Code idea in phishing website detection. We will explain what Detection-as-Code is and practical tips. This talk doesn't focus on "phishing" but Detection-as-Code so we wish you can have some takeaways!
## Who Are We?
- Manabu Niseki: A Botconf OBTS and JSAC speaker. A jack of all trades, master of none. A forever V3 climber.
- Dekai Wah: Dekai (Wally) is a Security engineer at LY Corporation’s Trust & Safety team.
## Background
Recently there were spikes on LINE themed phishing website attacks in Taiwan.
For example:
- [詐騙無所不在！朋友突傳 LINE「幫我投票」一登入秒被盜 下秒親友全遭殃 ](https://tw.news.yahoo.com/%E8%A9%90%E9%A8%99%E7%84%A1%E6%89%80%E4%B8%8D%E5%9C%A8-%E6%9C%8B%E5%8F%8B%E7%AA%81%E5%82%B3line-%E5%B9%AB%E6%88%91%E6%8A%95%E7%A5%A8-%E7%99%BB%E5%85%A5%E7%A7%92%E8%A2%AB%E7%9B%9C-%E4%B8%8B%E7%A7%92%E8%A6%AA%E5%8F%8B%E5%85%A8%E9%81%AD%E6%AE%83-101055130.html)
- [小心！LINE 詐騙新招來了 1 步驟帳號秒被盜](https://tw.news.yahoo.com/%E5%B0%8F%E5%BF%83-line%E8%A9%90%E9%A8%99%E6%96%B0%E6%8B%9B%E4%BE%86%E4%BA%86-1%E6%AD%A5%E9%A9%9F%E5%B8%B3%E8%99%9F%E7%A7%92%E8%A2%AB%E7%9B%9C-040051437.html)
- [可幫我的狗狗投一票？私訊收這千萬別點「秒被盜帳號」](<(https://tw.news.yahoo.com/%E5%8F%AF%E5%B9%AB%E6%88%91%E7%9A%84%E7%8B%97%E7%8B%97%E6%8A%95-%E7%A5%A8-%E7%A7%81%E8%A8%8A%E6%94%B6%E9%80%99%E5%8D%83%E8%90%AC%E5%88%A5%E9%BB%9E-%E7%A7%92%E8%A2%AB%E7%9B%9C%E5%B8%B3%E8%99%9F-072015827.html)>)
We've detected & responsed 396 cases from 2024/01/01 to 2024/07/16. (including non-TW regions.)
**Heatmap of /8 Spaces Across IPv4** (including multiple occurrences)
**Parent-domain tree** (including multiple occurrences)
This talk explains how we detect & response them with detection engineering in mind.
This experience gives us a unique opportunity of designing pipelines, a Detection-as-Code pipeline and a detection & response pipeline, from scratch.
So we are here to share what we've learned!
::: info
Note that we also do other countermeasures against account take over but we don't talk about them in this talk.
:::
## Topics
- What is Detection-as-Code?
- Detection-as-Code pipeline
- Detection & response pipeline
- Lessons learned
::: info
- Detection-as-Code pipeline is about how to develop & manage detection rules.
- Detection & response pipeline is about how to operate detection rules.
:::
**Overview**
```mermaid
flowchart LR
Repository --> PullRequest[Pull request]
PullRequest --> Lint
PullRequest --> Test
subgraph CI["CI (GitHub Actions)"]
Lint
Test
end
Test --> Review
Lint --> Review
subgraph CD
Deployment
end
Review --> Deployment
Deployment --> Detection
subgraph Collection
VirusTotal --> Airflow[<img src='https://pbs.twimg.com/profile_images/1176455256869412866/Xu7llkL3_400x400.png' /> Airflow]
urlscan.io --> Airflow
Customer --> Airflow
end
Airflow --> FastAPID
subgraph Detection
FastAPID[<img src='https://pbs.twimg.com/profile_images/1754249231295029248/UYU70YQr_400x400.jpg' /> FastAPI] --> Playwright[<img src='https://playwright.dev/python/img/playwright-logo.svg' /> Playwright]
Playwright --> OPA[<img src='https://pbs.twimg.com/profile_images/888167838376013824/w5JtfUh1_400x400.jpg'/> OPA Rego]
OPA --> OpenSearch[<img src='https://community.appian.com/cfs-file/__key/communityserver-blogs-components-weblogfiles/00-00-00-00-50/opensearch.png'/> OpenSearch]
end
subgraph Monitoring
Sentry
end
OPA --> Sentry
OPA --> Case[Case Management]
Case --> FastAPIR
subgraph Response
FastAPIR[<img src='https://pbs.twimg.com/profile_images/1754249231295029248/UYU70YQr_400x400.jpg' /> FastAPI] --> Whois
FastAPIR --> RDAP
Whois --> Sigma[<img src='https://sigmahq.io/images/logo_light.svg' /> Sigma]
RDAP --> Sigma
Sigma --> Registrar
Sigma --> HostingProvider[Hosting provider]
Sigma --> Blocklist[Blocklist]
end
```
## What is Detection-as-Code?
Detection-as-Code (DaC) is a concept advocated by David Burkett (@signalblur) and Brandon Poole in 2019.
> As a result, software engineers turned to source code management (SCM) tooling, such as GitHub — used in conjunction with modern continuous integration/continuous deployment (CI/CD) pipeline tooling — which allows them to address the issues of version control, testing, and change control. These tools have the added benefit of being inexpensive, well documented and supported, and in many cases have existing integrations into other applications.
>
> Using the methodology defined in this paper, an organization can implement a Detector as Code strategy allowing security operations teams to benefit from the technologies and methodologies built and refined by their engineering counterparts.
> --- [Detectors as Code (Soteria Cybersecurity)](https://blog.soteria.io/detectors-as-code-b33e63baa2f0)
## Why Detection-as-Code?
```mermaid
mindmap
VCS
Pull request
Review
CI
Test
Lint
CD
Deployment
```
To make detection rules more solid through tests, lints and collaborations.
- Version Control System (VCS):
- Versioned rules allow easy tracking/rollback of changes.
- Pull request (PR) based development:
- A PR is a good way to encourage among a team to discuss/share changes. It will improve rules through collaboration.
- CI/CD:
- (Good) CI/CD assures quality & consistency and reduces operation costs.
- CI: unit tests, integration/E2E tests & lints.
- CD: automatic deployment.
## Detection-as-Code Pipeline
```mermaid
flowchart LR
Modeling --> Development
Development --> Test
Development --> Lint
Review --> Deployment
subgraph CI
Test
Lint
end
Test --> Review
Lint --> Review
subgraph CD
Deployment
end
```
### Modeling
- Define an event/log schema. Optional in many cases.
### Development
- Create a new rule (or modify an existing rule) via a PR.
### Test
- Run tests in CI. (e.g. CircleCI, GitHub Actions, etc.)
(including lint jobs)
### Lint
- Force to follow conventions. (e.g. how many spaces in an indent, etc.)
- Find common gotchas. (e.g. a `detection` is not referenced in `condtion` in Sigma.)
### Review
- Encourage collaboration & information sharing.
### Deployment
- Automatically deploy changes by CD. (e.g. CircleCI, GitHub Actions, etc.)
## Detection & Response Pipeline
```mermaid
flowchart LR
Collection --> Detection --> Response
```
- Data collection:
- Collect logs/events from sensors.
- Detection:
- Classify incoming logs/events.
- Response
- Do something based on the detection.
### Collection
```mermaid
flowchart LR
VirusTotal --> Airflow --> Detection
urlscan.io --> Airflow
Customer["Customer support"] --> Airflow
```
Collect data (URLs) from:
- VirusTotal
- urlscan.io
- Customer support
by using [Airflow](https://airflow.apache.org/).
::: info
You may wonder why don't solely relying on urlscan.io. It's because:
- urlscan.io's criterias != our criterias. urlscan.io's verdict may have false positives from our point of view.
- Seed. urlscan.io's support is great. They are so quick. But there is a time difference (-7 hours from JST/KST) and DIY is faster in many cases.
:::
### Detection
```mermaid
flowchart LR
URL --> HAR
HAR --> EHAR["Enriched HAR"]
EHAR --> Rego
Rego --> IP{Is phishy?}
IP --> Response
```
- Crawl a URL & generate [HAR](http://www.softwareishard.com/blog/har-12-spec/) data by using [Playwright](https://playwright.dev/).
- Enirch the HAR data by using 3rd party & in-house services. (e.g. logo detection, geolocation data, etc.)
- Classify the data by Rego.
### Response
```mermaid
flowchart LR
Domain --> Registrar
IP["IP address"] --> HP["Hosting Provider"]
URL --> Blocklist["Blocklist (e.g., Google Safe Browsing)"]
URL --> Domain
URL --> IP
```
- Register a URL in a blocklist. (e.g. Google Safe Browsing)
- Report a domain/an IP address to a registrar/a hosting provider to take it down.
Automation is a key for success. So we automate blocklisting and semi-automate takedown request.
We cannot automate the takedown request because of a last mile problem. Some service providers only accepts a request from a web form and their form is protected by a captcha.
## Tips or Lessons We’ve Learned
### Data Modeling
I believe so because you cannot forge a good detection rule if there is no good feature in the data model.
Tips for good data modeling are:
- Don't reinvent the wheel (if you have a chance to define a model as you want):
- Stand on the shoulders of giants (use a standard).
- A standard is a battle tested and it has higher coverage than your own invented model in most cases.
- Define features that a threat actor cannot change with ease. (e.g. has-logo, has-password-input-form, etc.)
### Development
- Don't rely on indicators (hashes, URLs, IP addresses, domains, etc.) as much as possible:
- It's very easy to change them.
- In other words, don't write a rule like [this](https://github.com/intezer/yara-rules/blob/master/RussianAPT.yar#L102-L222). (ref. https://x.com/cyb3rops/status/1810320638780846256)
- It's better to focus on TTPs:
- They are also subject to change but more durable.
- For example, checking which framework (Vue, React, jQuery, etc.) is used, checking what kind of data schema is used, etc.
- Select a right detection engine based on the data model.
#### Select a Right Detection Engine
I selected Rego because of the HAR based data model.
Let me explain why I selected Rego over YARA and Sigma.
##### Why Not YARA?
YARA is a flat earther. It's good to deal with binary data. But it's difficult or impossible to do a field based operation/comparison. In other words, how do you select `foo` in the following JSON by YARA?
```js
{
"foo": "bar"
}
```
This is a reason why there are:
- [JeromeHadorn/YARA-JSON](https://github.com/JeromeHadorn/YARA-JSON): YARA C Module for finding key and values in JSON files
- YARA-L: https://cloud.google.com/chronicle/docs/detection/yara-l-2-0-overview
##### Why Not Sigma?
Sigma is an one dimensional thinker. Sigma is bad at dealing with a list of objects.
```js
{
"users": [
{
"name": "foo"
},
{
"name": "bar"
}
]
}
```
It may depend on a backend, but in general, Sigma is not possible to select `users[*].name`.
There is a Sigma based phishing detection engine, [phish-report/IOK](https://github.com/phish-report/IOK), but it has the problem explained. For example, it cannot process a condition like whether it has a javascript file named `jquery.min.js` and it has a text `XXX` inside or not.
##### Why Rego?
> Rego was inspired by Datalog, which is a well understood, decades old query language. Rego extends Datalog to support structured document models such as JSON.
> Rego queries are assertions on data stored in OPA. These queries can be used to define policies that enumerate instances of data that violate the expected state of the system.
> --- [Policy Language (Open Policy Agent)](https://www.openpolicyagent.org/docs/latest/policy-language/)
Rego doesn't have problems I explained and it can be used as a general purpose classifier.
##### Rego 101
Rego is a Prolog/Datalog inspired language.
```rego=
# check whether it's example.com or not
package hitcon2024
import rego.v1
# use default keyword to set a default value
default is_example := false
is_example if {
# iterate over log.entries
some entry in input.log.entries
# the follwoing comparison expressions are grouped by AND
entry.request.url == "https://example.com/"
entry.response.content.mime_type == "text/html"
contains(entry.response.content.text, "Example Domain")
}
```
(https://play.openpolicyagent.org/p/YHYeNROleB)
You can check whether it has jQuery or not like the following:
```rego=
default has_jquery := false
has_jquery if count(jquery) > 0
javascripts := [entry |
some entry in input.log.entries
entry.response.content.mimeType in ["application/javascript", "text/javascript"]
]
jquery := [javascript |
some javascript in javascripts
contains(javascript.request.url, "jquery")
contains(javascript.response.content.text, "(c) OpenJS Foundation and other contributors | jquery.org/license")
]
```
##### Who Uses Rego? (In Terms of Detection Engineering)
- [trufflesecurity/logwarden](https://github.com/trufflesecurity/logwarden): Process all of your GCP audit logs against OPA Rego policies and alert for violations in real-time.
- [cisagov/ScubaGear/](https://github.com/cisagov/ScubaGear/): Automation to assess the state of your M365 tenant against CISA's baselines
- [m-mizutani/alertchain](https://github.com/m-mizutani/alertchain): Simple SOAR (Security Orchestration, Automation and Response) framework integrated with OPA/Rego
- [GoogleCloudPlatform/security-response-automation](https://github.com/GoogleCloudPlatform/security-response-automation): Take automated actions against threats and vulnerabilities. (Archived)
- [Detection Engineering and SOAR at Mercari](https://engineering.mercari.com/en/blog/entry/20220513-detection-engineering-and-soar-at-mercari/)
##### Appendix: YARA vs. Sigma vs. Rego
| Engine | Good at | Bad at |
| ------ | -------------------------------------------- | --------------- |
| YARA | Binary | Structured data |
| Sigma | Flat structured data / consecutive datasets | Non-flat structured data, Binary |
| Rego | Structured data | Consecutive datasets, Binary |
##### Appendix: Rego Performance
Tested with the following conditions:
- OPA v0.67.0
- 15+ policies
**example.com (13KB)**
```
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v1/data 3051 0(0.00%) | 3 2 25 3 | 256.02 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 3051 0(0.00%) | 3 2 25 3 | 256.02 0.00
Response time percentiles (approximated)
Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST /v1/data 3 3 4 4 4 4 5 6 21 25 25 3051
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
Aggregated 3 3 4 4 4 4 5 6 21 25 25 3051
```
**github.com (6.7MB)**
```
Type Name # reqs # fails | Avg Min Max Med | req/s failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST /v1/data 202 0(0.00%) | 276 269 353 270 | 3.38 0.00
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
Aggregated 202 0(0.00%) | 276 269 353 270 | 3.38 0.00
Response time percentiles (approximated)
Type Name 50% 66% 75% 80% 90% 95% 98% 99% 99.9% 99.99% 100% # reqs
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST /v1/data 270 280 280 280 280 290 300 320 350 350 350 202
--------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------
Aggregated 270 280 280 280 280 290 300 320 350 350 350 202
```
##### Appendix: Why Not a Generic Programming Language?
It's a very good question and my answer is it's up to you.
But I could argue some points why I prefer a detection (policy) language to a generic programming language:
###### Increasing Complexity
> While some complex detection use cases may require the flexibility of a full programming language, relying solely on languages like Python can unnecessarily increase complexity, make tuning much more difficult, raise the level of effort for other detection engineers on a team to understand, and increase the risk of errors and false negatives. It can also exclude skilled detection engineers who may not have programming expertise, making an already difficult-to-fill role even harder to staff.
> --- [A Five Year Retrospective on Detection as Code (Magonia Research)](https://www.magonia.io/blog/five-year-retrospective-detection-as-code)
In other words, a detection engine (Rego, YARA, etc.) can only evaluate an input. Nothing more than that. Which gives you a sharp focus on detection.
###### Data Idempotency
It depends on how you write the code but, in general, it's very easy to lose the data idempotency.
::: info
> An API call or operation is idempotent if it has the same result no matter how many times it's applied.
>
> --- [Idempotency (Square Developer)](https://developer.squareup.com/docs/build-basics/common-api-patterns/idempotency)
:::
```py
def rule_1(data: dict):
data["foo"] = "bar"
...
def rule_2(data: dict):
...
detections = [rule_1(data), rule_2(data), ...]
```
Using a dedicated detection language assures the data idempotency. In other words, a detection engine cannot interfere data in Python.
###### Isolated Runtimes
We use Python in a main application and the Python app queries OPA Rego to classify the data.
Thus we can update the Python app and the detection engine (Rego) individually. There is no need to update them together. It's cost effective.
```yaml
name: Deploy
on:
deploy-backend:
branches: [main]
paths:
- "backend/**"
deploy-opa-rego:
branches: [main]
paths:
- "policies/**.rego"
jobs:
deploy-backend: ...
deploy-opa-rego: ...
```
Additionally, the isolation can make the pipeline durable. A typo in Python code breaks the entire pipeline but a typo in Rego just breaks a rule itself.
### Test
Write tests, tests and tests.
- Unit tests for:
- Covering edge cases.
- It's easy to synthesize data for testing by using a property based testing library such as [HypothesisWorks/hypothesis](https://github.com/HypothesisWorks/hypothesis).
- Integration/E2E tests for:
- Assuring the integration of (3rd party) modules.
- Tests for detecting schema drift.
#### Unit Test
##### Rego
Rego provides test functionality by default.
**example_test.rego**
```rego=
package policy_test
import data.policy.allow
test_with_testdata {
allow with input as data.testdata.input1
not allow with input as data.testdata.input2
}
```
**input.json**
```js
{
"testdata": {
"input1": {
"my_input": "somedata"
},
"input2": {
"my_input": "someotherdata"
}
}
}
```
(ref. https://github.com/open-policy-agent/opa/issues/3026)
```bash
opa test example_test.rego
```
##### YARA
[VirusTotal/yara-x](https://github.com/VirusTotal/yara-x) provides clean API for testing.
```py
import yara_x
rules = yara_x.compile('''
rule test {
strings:
$a = "foobar"
condition:
$a
}''')
results = rules.scan(b"foobar")
assert results.matching_rules[0].identifier == "test"
assert results.matching_rules[0].patterns[0].identifier == "$a"
assert results.matching_rules[0].patterns[0].matches[0].offset == 0
assert results.matching_rules[0].patterns[0].matches[0].length == 6
```
(ref. https://virustotal.github.io/yara-x/docs/api/python/)
More practically, https://github.com/kevoreilly/CAPEv2/blob/722aced01eb7fbe976edb3ce372895deb0fb2106/tests/test_yara.py#L70-L83.
##### Sigma
It's a bit tricky because Sigma is a format. Sigma is not a detection engine. But you can test a Sigma rule with [ninoseki/azuma](https://github.com/ninoseki/azuma) (forked from [CybercentreCanada/pysigma](https://github.com/CybercentreCanada/pysigma)).
```py
from azuma import Rule
rule = Rule.model_validate_yaml(
"""
title: test
detection:
foo:
- bar
condition: foo
logsource:
category: test
"""
)
assert rule.match({"foo": "bar"}) is True
assert rule.match({"foo": "..."}) is False
```
Alternatively, you can consider using [bradleyjkemp/sigma-test](https://github.com/bradleyjkemp/sigma-test).
##### Appendix: Property Based Testing
> Property-based testing is an approach that involves specifying statements that should always be true, rather than relying on specific examples. It enables you to test functions across a large number of inputs with fewer tests.
> --- [Testing across a Large Number of Inputs with Property-Based Testing (InfoQ)](https://www.infoq.com/news/2023/07/property-based-testing/)
```py
import pytest
from hypothesis import given, settings
from hypothesis.provisional import urls
@pytest.fixture(scope="session")
def data() -> dict:
return ...
@given(urls())
@settings(max_examples=100)
def test_something(data: dict, url: str):
data["url"] = url
assert do_something(data) == ...
```
I don't say use this approach in all the tests. But it's useful for sometimes or many times not always.
#### Integration Test
In our case, we can easily write integration tests by using [csernazs/pytest-httpserver](https://github.com/csernazs/pytest-httpserver).
```py=
from pytest_httpserver import HTTPServer
def test_integration(httpserver: HTTPServer):
httpserver.expect_request("/foo").respond_with_response(...)
snapshot = SnapshotFactory().call(httpserver.url_for("/foo"))
assert snapshot.policies = [...]
```
Below are good references to perform integration tests for endpoint, SIEM, etc.:
- [splunk/attack_range](https://github.com/splunk/attack_range): A tool that allows you to create vulnerable instrumented local or cloud environments to simulate attacks against and collect the data into Splunk
- [redcanaryco/atomic-red-team](https://github.com/redcanaryco/atomic-red-team): Small and highly portable detection tests based on MITRE's ATT&CK.
- [mitre/caldera](https://github.com/mitre/caldera): Automated Adversary Emulation Platform
- [3CORESec/Automata](https://github.com/3CORESec/Automata): Automatic detection engineering technical state compliance
- [threat-punter/detection-as-code-example](https://github.com/threat-punter/detection-as-code-example): A POC to implement Detection-as-Code with Terraform and Sumo Logic.
- [Intel as Code - Building a Threat Informed Security Organization (YouTube)](https://www.youtube.com/watch?v=5b0_41WhHNY)
#### Appendix: Schema Drift
> Schema drift is the case where your sources often change metadata. Fields, columns, and, types can be added, removed, or changed on the fly. Without handling for schema drift, your data flow becomes vulnerable to upstream data source changes.
>
> --- [Schema drift in mapping data flow (Microsoft Learn)](https://learn.microsoft.com/en-us/azure/data-factory/concepts-data-flow-schema-drift)
We have testing data in two places `tests/fixtures/samples/*/*.json` (fixtures for pytest) and `policies/*/*.json` (fixtures for Rego).
It will be a disaster if there is a fixture which has a driffted field. It gives a wrong input for a rule during testing and a test will pass even a rule does not work in production.
We prevent a such case by testing fixtures by using [pydantic/pydantic](https://github.com/pydantic/pydantic).
```py
# schema
from pydantic import BaseModel
class Snapshot(BaseModel):
log: ... # HAR log
...
# test
import pytest
def get_keys(data: dict) -> list[str]:
# returns all the keys in a given dict
...
@pytest.fixture
def data() -> dict:
with open("/path/to/json", "r") as f:
return json.loads(f.read())
def test_data(data: dict):
validated = Snapshot.model_validate(data).model_dump()
# a set of keys of the fixture data
data_keys = set(get_keys(data))
# a set of keys of the validated data
# (= keys/fields not defined in the schema are excluded)
validated_keys = set(get_keys(validated))
# this assertion ensures that there is no extra key in the fixture data
assert len(data_keys - validated_keys) == 0
```
#### Appedix: TDD or TDDD?
> Typically, the detection engineering process at a high level involves receiving threat intelligence, writing a detection rule, and then creating a test to match the intended detection logic. This process can seem backward, and it often is. There are several reasons for this, such as the difficulty of setting up a safe test environment and the lack of a structured, repeatable way to execute the tests used to trigger a detection. Tooling in this area is only now maturing with open-source projects like Mitre’s Caldera and Red Canary’s Atomic Red Team, as well as other commercial tools in this space. Sometimes, tests are retroactively built to match the detection logic rather than the source intelligence, which, while similar, isn’t necessarily the same. For example an analyst may misunderstand the behavior in the intel, and build their test to match the misunderstood behavior.
>
> These issues can unintentionally hinder detection engineering teams. However, we can adopt lessons from other disciplines, such as Test-Driven Development, to improve this process. Test-Driven Development is a software development methodology where tests are written before the code itself. The same process should be adopted by detection teams when at all possible.
> --- [A Five Year Retrospective on Detection as Code (Magonia Research)](https://www.magonia.io/blog/five-year-retrospective-detection-as-code)
I agree with this opinion but I would say it would be Test-Data-Driven Developement (TDDD). Preparing data (logs, samples, etc.) first and write a detection rule for it.
You can skip manually writing a test by accepting some conventions. For example, we stored sample data with the following convention:`./tests/fixtures/samples/{POLICY}/*.json`.
Then you don't need to write a new test but you need define a new sample data and a detection rule.
```py
import glob
import json
import pytest
from pydantic import BaseModel
class Sample(BaseModel):
path: str
policy: str
data: dict
def get_samples() -> list[Sample]:
samples: list[Sample] = []
for path in glob.glob("tests/fixtures/samples/*/*.json"):
with open(path) as f:
data: dict = json.loads(f.read())
policy = path.split("/")[-2]
samples.append(Sample(path=path, policy=policy, data=data))
return samples
@pytest.fixture(params=get_samples())
def sample(request: pytest.FixtureRequest) -> Sample:
return request.param
def test_sample(sample: Sample, opa: OPA):
assert sample.policy == opa.call(sample.data)
```
### Lint
- Always use a linter.
- Tame a linter to make it fit with your hands.
#### Rego
- [StyraInc/regal](https://github.com/StyraInc/regal): Regal is a linter for Rego, with the goal of making your Rego magnificent!
```rego
// test.rego
package policy
import rego.v1
allow if input.foo == "bar"
allow if input.bar == "baz"
# we already covered this!
allow if input.foo == "bar"
```
```bash
$ regal lint test.rego
Rule: duplicate-rule
Description: Duplicate rule found at line 10
Category: bugs
Location: test.rego:5:1
Text: allow if input.foo == "bar"
Documentation: https://docs.styra.com/regal/rules/bugs/duplicate-rule
1 file linted. 1 violation found.
```
#### Sigma
- [SigmaHQ/sigma-cli](https://github.com/SigmaHQ/sigma-cli): The Sigma command line interface based on pySigma
- [bradleyjkemp/sigmafmt](https://github.com/bradleyjkemp/sigmafmt): An opinionated formatter/linter for Sigma rules
- [certeu/droid](https://github.com/certeu/droid): A pySigma wrapper to manage detection rules
- [ninoseki/azuma](https://github.com/ninoseki/azuma): Yet another Sigma library for Python
**`sigma-cli`**
```yaml
# test.yaml
title: simgma-cli demonstration
status: experimental
description: can sigma-cli detects common gotchas?
logsource:
category: process_creation
product: windows
detection:
selection:
Image:
- svchost.exe
foo:
bar: baz
condition: selection
```
```bash
$ sigma check -ei test.yml
...
=== Summary ===
Found 0 errors, 0 condition errors and 5 issues.
No rule errors found.
No condition errors found.
Validation issue summary:
+-------+--------------------------+----------+--------------------------------------------------------------+
| Count | Issue | Severity | Description |
+-------+--------------------------+----------+--------------------------------------------------------------+
| 1 | IdentifierExistenceIssue | MEDIUM | Rule has no identifier (UUID) |
| 1 | DateExistenceIssue | MEDIUM | Rule has no date |
| 1 | FilenameLenghIssue | HIGH | Rule filename is too short or long |
| 1 | DanglingDetectionIssue | HIGH | Rule defines detection that is not referenced from condition |
| 1 | LevelExistenceIssue | MEDIUM | Rule has no level |
+-------+--------------------------+----------+--------------------------------------------------------------+
Check failure
```
#### YARA
- [CybercentreCanada/CCCS-Yara](https://github.com/CybercentreCanada/CCCS-Yara): YARA rule metadata specification and validation utility / Spécification et validation pour les règles YARA
- [YARAHQ/yara-forge](https://github.com/YARAHQ/yara-forge): Automated YARA Rule Standardization and Quality Assurance Tool
**CCCS-Yara**
```yara
rule silent_banker : banker {
meta:
description = "This is just an example"
threat_level = 3
in_the_wild = true
strings:
$a = {6A 40 68 00 30 00 00 6A 14 8D 91}
$b = {8D 4D B0 2B C1 83 C0 27 99 6A 4E 59 F7 F9}
$c = "UVODFRYSIHLNWPEJXQZAKCBGMT"
condition:
$a or $b or $c
}
```
```bash
$ yara_validator -v test.yar
____ ____ ____ ____ __ __ _ ____ _
/ ___/ ___/ ___/ ___| \ \ / // \ | _ \ / \
| | | | | | \___ \ \ V // _ \ | |_) | / _ \
| |__| |__| |___ ___) | | |/ ___ \| _ < / ___ \
\____\____\____|____/ |_/_/ \_\_| \_\/_/ \_\
🍩 Invalid Rule File: test.yar
- Errors:
- silent_banker:
- id: ⚙️ Missing metadata that could have been generated with the -i or -c flag for the cli
- fingerprint: ⚙️ Missing metadata that could have been generated with the -i or -c flag for the cli
- version: ⚙️ Missing metadata that could have been generated with the -i or -c flag for the cli
- modified: ⚙️ Missing metadata that could have been generated with the -i or -c flag for the cli
- status: Missing required metadata
- sharing: Missing required metadata
- source: Missing required metadata
- author: Missing required metadata
- category: Missing required metadata
----------------------------------------------------------------------------
All .yara Rule files found have been passed through the CCCS Yara Validator:
Total Yara Rule Files to Analyze: 1
Total Valid CCCS Yara Rule Files: 0
Total Warning CCCS Yara Rule Files: 0
Total Invalid CCCS Yara Rule Files: 1
---------------------------------------------------------------------------
```
#### Lint With Git Hook
Optionally & additionally, I also recommend to use a Git hook manager such as [evilmartians/lefthook](https://github.com/evilmartians/lefthook) and run lint on pre-commit/pre-push.
```yaml=
# lefthook.yml
pre-commit:
commands:
# lint Rego policies with Regal
rego:
glob: "*.rego"
run: regal lint {staged_files}
# validate Sigma rules with sigma-cli
sigma:
root: "/path/to/sigma"
glob: "*.{yaml,yml}"
run: sigma check -ei {staged_files}
# lint YARA ruels with CCCS-Yara
yara:
glob: "*.{yara,yar}"
run: yara_validator -v {staged_files}
```
### Deployment
Automate! I have nothing to add more.
### Response
Automate the reponse tasks as much as possible.
For example, we autoamte like the following:
- Use Google Safe Browsing's hidden/undocumented API for reporting:
- https://github.com/chromium/suspicious-site-reporter/blob/725348a01adcf6c8966b65372f84b95521bafb0d/extension/background.js#L148-L162
- Use a tool to find where to report:
- [certsocietegenerale/abuse_finder](https://github.com/certsocietegenerale/abuse_finder): Find abuse contacts for observables
- [bradleyjkemp/abwhose](https://github.com/bradleyjkemp/abwhose): The simplest way to find how to report abusive domains
- [ninosei/abuse_whois](https://github.com/ninoseki/abuse_whois): Yet another way to find where to report an abuse
We use `abuse_whois`, a Sigma based tool because the data it handles is "flat".
```js
{
"admin_name": "DATA REDACTED",
"admin_organization": "DATA REDACTED",
"admin_email": "https://domaincontact.cloudflareregistrar.com/hitcon.org",
"admin_address": "DATA REDACTED, DATA REDACTED, DATA REDACTED, DATA REDACTED, DATA REDACTED, DATA REDACTED",
"admin_phone": "tel:DATA REDACTED ext DATA REDACTED",
"admin_fax": "tel:DATA REDACTED ext DATA REDACTED",
"billing_name": "DATA REDACTED",
"billing_organization": "DATA REDACTED",
"billing_email": "https://domaincontact.cloudflareregistrar.com/hitcon.org",
"billing_address": "DATA REDACTED, DATA REDACTED, DATA REDACTED, DATA REDACTED, DATA REDACTED, DATA REDACTED",
"billing_phone": "tel:DATA REDACTED ext DATA REDACTED",
"billing_fax": "tel:DATA REDACTED ext DATA REDACTED",
"registrant_name": "DATA REDACTED",
"registrant_organization": null,
"registrant_email": "https://domaincontact.cloudflareregistrar.com/hitcon.org",
"registrant_address": "DATA REDACTED, DATA REDACTED, DATA REDACTED, Taiwan, DATA REDACTED, TW",
"registrant_phone": "tel:DATA REDACTED ext DATA REDACTED",
"registrant_fax": "tel:DATA REDACTED ext DATA REDACTED",
"status": [
"clienttransferprohibited"
],
"domain_name": "hitcon.org",
...
}
```
(Note: it uses RDAP & Whois. But it parses & transforms the data into the flat format by using [pogzyb/asyncwhois](https://github.com/pogzyb/asyncwhois))
### Etc.
#### Have a Feedback Loop
In other words, have a way to know false negatives/positives.
We use [getsentry/sentry](https://github.com/getsentry/sentry) for error/issue tracking. We gets an issue if an input from trusted sources is classified as not phishy.
#### Documentation Through Metadata
A detection rule should have a corresponding document. But managing docs outside of VCS can be tricky (e.g. managing docs in Confluence). Such docs are easy to be outdated. So I recommend to document a rule through metadata. It ensures the [single source of truth](https://en.wikipedia.org/wiki/Single_source_of_truth).
##### Rego
Rego can have `metadata` block to make it self explanatory.
```rego
# METADATA
# title: My rule
# description: A rule that determines if x is allowed.
# authors:
# - John Doe <john@example.com>
# entrypoint: true
allow if {
...
}
```
(ref. https://www.openpolicyagent.org/docs/latest/policy-language/#metadata)
##### YARA
YARA also can have `meta` block.
- https://virustotal.github.io/yara-x/docs/writing_rules/anatomy-of-a-rule/#metadata
But there is no official recommendation what should be defined in `meta`. So I recommend to follow CCCS-Yara's spec.
- Spec: https://github.com/CybercentreCanada/CCCS-Yara/blob/master/CCCS_YARA.yml
- Example: https://github.com/CybercentreCanada/assemblyline-service-jsjaws/blob/da935bc7981ffea0a2d2529ff03dacd77f532e48/yara/js_obfuscator_io.yara#L3-L18
##### Sigma
Sigma has metadata fields (e.g. `title`, `description`, `references`, etc.) for documtation.
- https://sigmahq.io/docs/basics/rules.html#available-sigma-metadata-fields
- Example: https://github.com/SigmaHQ/sigma/blob/master/rules-threat-hunting/linux/file/file_event/file_event_lnx_python_path_configuration_files.yml#L1-L22
## Wrap-up / Key Takeaways
- In short, DaC is a practice that applies DevOps best practices into detection engineering.
- Data modeling is the king rules all.
- Write tests, tests and tests to get self or team confidence.
- You may feel that writing tests is boring. A TDDD approach makes it easy.
- Make things verbose & explicit:
- "as Code" approach increases transparency in general. Adding metadata in detection rules makes them event better. Adhere to the single source of truth.
- Last but not least, automate all the things!
## Acknowledgements
Thanks to `@__Thanat0s__` and `@sepi140` for reviewing this paper.
Sign in with Wallet
Connect another wallet