---
title: "Observability and SRE"
chapter: "13"
---

# Observability and SRE

Observability answers what users experience, why it happens, and what to do.

## Signals

Collect metrics, logs, traces, events, audit logs, profiles, and business
outcomes. GKE integrates with Cloud Logging and Cloud Monitoring. Managed
Service for Prometheus supports Prometheus metrics and queries. OpenTelemetry
provides portable instrumentation.

## SLO first

Define service-level indicators such as successful request ratio and latency.
Set an SLO and error budget. Alert on fast or sustained error-budget burn rather
than every transient metric. Infrastructure alarms should be actionable causes.

## Kubernetes signals

Watch unschedulable Pods, restart/OOM rate, readiness, CPU throttling, memory,
node pressure, HPA limits, control-plane/API errors, workload latency, queue
age, and persistent-volume behavior.

## Cardinality and cost

Unbounded labels such as user ID create expensive, unusable metrics. Structure
logs, set retention, sample traces deliberately, and keep labels bounded.

## Runbooks

Every urgent alert needs an owner, impact statement, dashboards, safe diagnosis,
mitigation, escalation, and verification. Automate repeatable safe actions, but
keep audit and rollback.

## Learning loop

Blameless postmortems identify contributing conditions and concrete follow-up.
Track action ownership. Reliability improves when incident learning changes
code, policy, tests, and platform defaults.