In production, complex infrastructure (both hardware and software) is often like a black box. When things start to go wrong, it is critical to know early and to have enough relevant information to pinpoint the issue quickly.
Observability lets us understand a system from the outside, by letting us ask questions about that system without knowing its inner workings. Furthermore, allows us to easily troubleshoot and handle novel problems (i.e. “unknown unknowns”), and helps us answer the question, “Why is this happening?”
https://opentelemetry.io/docs/concepts/observability-primer/
"There was no monitoring, there was no dashboard, there was nothing, there was no place to look to see if the site was working today or not, except for CNN"
– Mikey Dickerson on being brought in to help fix healthcare.gov
(don't do that. have monitoring.)
Metrics, Logs, and Traces
Numeric representations of data measured over intervals of time.
Easy to visualize and correlate, efficient to generate, collect and store
e.g:
An immutable, timestamped record of discrete events that happened over time.
Doesn't require much explanation.
Logs are great, but there can be "too much of a good thing", it is difficult to piece together a failure from a mountain of logs in real time. And they don't provide much context.
An end-to-end, detailed construction of an entire action, which can sometimes cross service boundaries.
Traces are a bit like logs on steroids. They are typically associated with a particular user request or transaction, and include data about significant steps during the processing of the request, including potentially hops across different services.
More structure available than logs
server{host="localhost", port=8080}
0ms INFO starting
300ms INFO listening
conn{peer_addr="82.9.9.9", port=42381}
0ms DEBUG connected
300ms DEBUG message received, length=2
conn{peer_addr="8.8.8.8", port=18230}
300ms DEBUG connected
conn{peer_addr="82.9.9.9", port=42381}
600ms WARN weak encryption requested, algo="xor"
901ms DEBUG response sent, length=8
901ms DEBUG disconnected
1502ms WARN internal error
1502ms INFO exit
It's easier to go from a structured collection of correlated events (a trace) to flat uncorrelated logs, than to go from a set of uncorrelated log events to a trace. Thus sometimes logging functionality provided by a tracing library is preferable since it works both ways.
Most developers are familiar with the concept of "profiling" - a way of automatically instrumenting a program so that performance can be measured in a fine-grained way, often down to individual lines or functions.
A mental model of a tracing is thus:
"OpenTelemetry is a collection of tools, APIs, and SDKs for instrumenting, generating, collecting, and exporting telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior."
The "telemetry" in "opentelemetry" is closer to "a user monitoring their application" than what the Pulp team refers to as "telemetry" which is "automated usage feedback provided by". The overlap is very limited.
The project is part of the CNCF, the same organization behind Kubernetes.
Opentelemetry is rapidly becoming the standard set of libraries (across many programming languages) you use to get this functionality. The ecosystem is largely consolidating around opentelemetry.
Python, Ruby, C++, Go, .NET, Java, Javascript, Rust, Swift, PHP, Erlang/Elixr
Second most active CNCF project, behind Kubernetes
Applications are getting more distributed. Many applications (including Pulp) are spread multiple services, even if they are not architected as "micro-" services.
And even monolithic applications are often reliant on "services" such as a database, a Redis cache, etc.
Monitoring a distributed application presents unique and difficult challenges, and the ecosystem strongly benefits from the network effects of building standards.
The consolidation around opentelemetry means that collection backends and other tools can be swapped without needing to re-instrument code using entirely new libraries.
Opentelemetry became the standard toolset for tracing early on, and is expanded from there to metrics and logs.
API: in charge of gathering the telemetry and all the data that is part of it
SDK: what gets this data out of the current observed process to another entity for analysis.
Collector:
"So why choose the OTel Collector when you can send directly to Jaeger, Prometheus, or the console? Because of flexibility:"
Prometheus is a time-series metrics monitoring tool, it comes with a data storage solution and some analytics capability.
Opentelemetry is a vendor-agnostic set of specifications and libraries. It doesn't do analytics or storage, you need to choose a backend for that. But the data can be exported into almost any backend, including Prometheus!
Metrics are usually about aggregating data.
An aggregation is a technique whereby a large number of measurements are combined into either exact or estimated statistics about metric events that took place during a time window.
Common aggregations: sum, count, average, histogram
Examples from:
https://uptrace.dev/opentelemetry/python-metrics.html#getting-started
https://uptrace.dev/opentelemetry/metrics.html
counter: a value that is summed over time – you can think of this like an odometer on a car; it only ever goes up.
measure: a value that is aggregated over time. This is more akin to the trip odometer on a car, it represents a value over some defined range.
observer: captures a current set of values at a particular point in time, like a fuel gauge in a vehicle.
Synchronous instruments are invoked together with operations they are measuring. For example, to measure the number of requests, you can call counter.Add(ctx, 1) whenever there is a new request.
Counter
counter = meter.create_counter(
name="total_requests", description="TODO", unit="1",
)
counter.add(1)
counter.add(10)
e.g. number of requests, bytes read/written
Bidirectional counter
counter = meter.create_up_down_counter(name="updown_counter", description="TODO")
while True:
if random.random() >= 0.5:
counter.add(+1)
else:
counter.add(-1)
time.sleep(1)
e.g. number of active connections
Histogram
histogram = meter.create_histogram(
name="some.prefix.histogram",
description="TODO",
unit="microseconds",
)
while True:
histogram.record(random.randint(1, 5000000), attributes={"attr1": "value1"})
time.sleep(1)
e.g. request duration
Asynchronous instruments (observers) periodically invoke a callback function to collect measurements. For example, you can use observers to periodically measure memory or CPU usage.
GaugeObserver
def callback():
# calculate current value on-demand
return [Measurement(random.random())]
gauge = meter.create_observable_gauge(
name="some.prefix.gauge_observer",
callback=callback,
description="TODO",
)
Non-additive values for which sum does not produce a meaningful correct result, e.g. resource utilization, error rate (percentage)
Metrics are automatically collected by opentelemetry in-process, aggregated & exported.
Unlike logs, the cost of metrics doesn’t increase in lockstep with user traffic or any other system activity that could result in a sharp uptick in data. Client-side aggregation reduces traffic.
"A Span represents a unit of work or operation. It tracks specific operations that a request makes, painting a picture of what happened during the time in which that operation was executed.
A Span contains name, time-related data, structured log messages, and other metadata (i.e. Attributes) to provide information about the operation it tracks."
A Trace records the progression of requests (made by an application or end-user) as they propagate across different services and components in our app (DB, data sources, queues, etc.)
Like ‘Call-stacks’ for distributed services. Traces represent a tree of spans connected in a child/parent relationship.
Without tracing, it is challenging to pinpoint the cause of performance problems in a distributed system.
We want any and all data relevant to a "span" to be available, so that we can draw conclusions when investigating the trace. Attributes are metadata attached to the span such as:
Once we create a span within our application, the exporter handles sending the data to our backend (e.g., in memory, Jaeger Tracing, or console output)
Live demo
PulpCon 2022