## Observability for Pentesters --- ### About Me - Ex-Pentester/IT Security person - Senior Security Advocate at Datadog - CIS Benchmark author, Docker and Kubernetes - Member of Kubernetes SIG-Security & CNCF TAG-Security --- ## Agenda - What is Observability? - Attacking Observability - Using Observability in pentests --- ## What is observability? > Observability lets us understand a system from the outside, by letting us ask questions about that system without knowing its inner workings. > Furthermore, it allows us to easily troubleshoot and handle novel problems (i.e. “unknown unknowns”), and helps us answer the question, “Why is this happening?” --- ## How does that work? - Logs - Metrics - Traces - Correlation! --- ## Basic architecture  --- ## Observability tools  --- ## Core open source observability tools  --- ##  Fluentd - Fluentd is an open source data collector for unified logging layer. - Newer versions (Fluent bit) also do metrics and traces. Note: From https://www.fluentd.org/ --- ##  Jaeger - Backend UI for Tracing Note: https://www.jaegertracing.io/ --- ##  Prometheus - Gathering and analysing metrics Note: Cortex, Thanos - High Availability Prometheus OpenMetrics - Project on prometheus data format. --- ##  OpenTelemetry - SDK and API for application tracing - Open Telemetry Collector --- ## Observability tool security --- ## A word about threat models > It is presumed that untrusted users have access to the Prometheus HTTP endpoint and logs. They have access to all time series information contained in the database, plus a variety of operational/debugging information. Note: https://prometheus.io/docs/operating/security/ https://www.jaegertracing.io/docs/1.57/security/ --- ## Encryption - Often not enabled by default, often self-signed certificates --- ## Authentication - Prometheus. Supported, but not enabled by default - Jaeger. No in-built Authentication - Suggested solution is a proxy server. Note: These are the two places where there's serices listening that can be connected to, neither of them have AuthN out of the box. Jaeger doesn't do TLS either, so it needs an external proxy. https://github.com/jaegertracing/jaeger/issues/4840 - No AuthN for Jaeger --- ## What can we get? - Metric Information - Trace Information - Log Information --- ## Prometheus - From prometheus Endpoints (9090/TCP, 9100/TCP) - Demo! Note: Because prometheus scrapes services, those services expose information, generally at a URL of /metrics. Whether the specific information is useful or not is rather situational. We can look at localhost:9090 and show the kinds of information in there which aren't *massively* useful http://127.0.0.1:9090/tsdb-status has some useful stats. Next http://127.0.0.1:9100/metrics we can CTRL-F for "device" and show how node metrics expose quite a lot of date Also 724K hosts on the internet with node exporter. https://search.censys.io/search?resource=hosts&sort=RELEVANCE&per_page=25&virtual_hosts=EXCLUDE&q=%22Node+Exporter%22 --- ## Trace Information - From Jaeger back ends (16686/TCP) - Demo! Note: Here we can show how to get traces from a Jaeger environment and also how there can be *interesting* information in there. Specifically pick a trace from the front-end web and we should be able to see session information. Only 1784 hosts with Jaeger UI https://search.censys.io/search?resource=hosts&sort=RELEVANCE&per_page=25&virtual_hosts=EXCLUDE&q=%22jaeger+UI%22 --- ## Using observability tools for security work --- ## Threat modelling ---  --- ## Pentesting ---  --- ## Conclusion - Observability tooling becoming increasingly common - Security defaults are often not strong - Some great potential use cases in pentesting and security --- ## Thanks! - E-Mail - rorym@mccune.org.uk - Mastodon - @raesene@infosec.exchange - Linkedin - https://www.linkedin.com/in/rorym/