Observability and SRE

Observability answers what users experience, why it happens, and what to do.

Signals

Collect metrics, logs, traces, events, audit logs, profiles, and business outcomes. GKE integrates with Cloud Logging and Cloud Monitoring. Managed Service for Prometheus supports Prometheus metrics and queries. OpenTelemetry provides portable instrumentation.

SLO first

Define service-level indicators such as successful request ratio and latency. Set an SLO and error budget. Alert on fast or sustained error-budget burn rather than every transient metric. Infrastructure alarms should be actionable causes.

Kubernetes signals

Watch unschedulable Pods, restart/OOM rate, readiness, CPU throttling, memory, node pressure, HPA limits, control-plane/API errors, workload latency, queue age, and persistent-volume behavior.

Cardinality and cost

Unbounded labels such as user ID create expensive, unusable metrics. Structure logs, set retention, sample traces deliberately, and keep labels bounded.

Runbooks

Every urgent alert needs an owner, impact statement, dashboards, safe diagnosis, mitigation, escalation, and verification. Automate repeatable safe actions, but keep audit and rollback.

Learning loop

Blameless postmortems identify contributing conditions and concrete follow-up. Track action ownership. Reliability improves when incident learning changes code, policy, tests, and platform defaults.