59 lines
2.1 KiB
Markdown
59 lines
2.1 KiB
Markdown
---
|
|
title: Monitoring and Observability
|
|
description: Core concepts behind monitoring, alerting, and observability for self-hosted systems
|
|
tags:
|
|
- monitoring
|
|
- observability
|
|
- operations
|
|
category: infrastructure
|
|
created: 2026-03-14
|
|
updated: 2026-03-14
|
|
---
|
|
|
|
# Monitoring and Observability
|
|
|
|
## Summary
|
|
|
|
Monitoring and observability provide visibility into system health, failure modes, and operational behavior. For self-hosted systems, they turn infrastructure from a black box into an environment that can be maintained intentionally.
|
|
|
|
## Why it matters
|
|
|
|
Without visibility, teams discover failures only after users notice them. Observability reduces diagnosis time, helps verify changes safely, and supports day-two operations such as capacity planning and backup validation.
|
|
|
|
## Core concepts
|
|
|
|
- Metrics: numerical measurements over time
|
|
- Logs: event records produced by systems and applications
|
|
- Traces: request-path visibility across components
|
|
- Alerting: notifications triggered by actionable failure conditions
|
|
- Service-level thinking: monitoring what users experience, not only host resource usage
|
|
|
|
## Practical usage
|
|
|
|
A practical starting point often includes:
|
|
|
|
- Host metrics from exporters
|
|
- Availability checks for critical endpoints
|
|
- Dashboards for infrastructure and core services
|
|
- Alerts for outages, storage pressure, certificate expiry, and failed backups
|
|
|
|
## Best practices
|
|
|
|
- Monitor both infrastructure health and service reachability
|
|
- Alert on conditions that require action
|
|
- Keep dashboards focused on questions operators actually ask
|
|
- Use monitoring data to validate upgrades and incident recovery
|
|
|
|
## Pitfalls
|
|
|
|
- Treating dashboards as a substitute for alerts
|
|
- Collecting far more data than anyone reviews
|
|
- Monitoring only CPU and RAM while ignoring ingress, DNS, and backups
|
|
- Sending noisy alerts that train operators to ignore them
|
|
|
|
## References
|
|
|
|
- [Prometheus overview](https://prometheus.io/docs/introduction/overview/)
|
|
- [Prometheus Alertmanager overview](https://prometheus.io/docs/alerting/latest/overview/)
|
|
- [Grafana documentation](https://grafana.com/docs/grafana/latest/)
|