To set up Prometheus on Kubernetes, you deploy the Prometheus server, configure it to discover Kubernetes endpoints via the Kubernetes API, and expose its UI via a Service or Ingress. The most common approach is using the Prometheus Operator or the kube-prometheus-stack Helm chart, which automates deployment, service discovery, and alerting rules.
Background: Why Prometheus on Kubernetes
Prometheus is an open-source monitoring and alerting toolkit that scrapes metrics from configured targets at regular intervals. Kubernetes is a container-orchestration platform where pods and services are ephemeral, making traditional monitoring (based on fixed IPs) impractical. Prometheus’s pull-based model and built-in service discovery make it a natural fit: it queries the Kubernetes API to find targets, then scrapes metrics from them. This integration is so common that Prometheus is now a Cloud Native Computing Foundation (CNCF) graduated project, and its Kubernetes service discovery is a core feature.
Core Components and How They Work
A full Prometheus setup on Kubernetes typically includes:
- Prometheus Server: Scrapes and stores metrics, evaluates alerting rules, and serves the UI/API.
- Service Discovery: Uses Kubernetes API to find pods, services, and nodes, updating the scrape targets dynamically.
- Alertmanager: Handles alerts generated by Prometheus, deduplicating and routing them to channels like email or Slack.
- Exporters: Small agents that expose metrics from applications that don’t natively support Prometheus format (e.g., node-exporter for host metrics, kube-state-metrics for Kubernetes objects).
- Grafana (optional): A visualization tool that queries Prometheus to create dashboards.
In a typical flow, Prometheus discovers targets via the Kubernetes API, scrapes metrics from them over HTTP, stores the data in a time-series database, and evaluates rules to trigger alerts. The kube-prometheus-stack Helm chart bundles all of these components, including default dashboards and alerting rules.
Choosing a Setup Method
There are three main ways to deploy Prometheus on Kubernetes, each with trade-offs:
- Helm chart (kube-prometheus-stack): The most popular choice. It installs Prometheus, Alertmanager, Grafana, node-exporter, and kube-state-metrics with sensible defaults. Recommended for most users because it’s quick and maintainable.
- Prometheus Operator: A Kubernetes operator that manages Prometheus instances via custom resources (e.g., Prometheus, ServiceMonitor). It’s more flexible but requires more manual configuration. The kube-prometheus-stack is built on top of this operator.
- Manual YAML manifests: Writing your own Deployment, Service, and ConfigMap from scratch. Gives full control but is time-consuming and error-prone. Best for learning or highly customized environments.
For a production setup, the Helm chart is the de facto standard because it includes pre-configured alerting rules and dashboards that work out of the box.
Step-by-Step Setup with Helm
Assuming you have a running Kubernetes cluster and kubectl configured, here’s a typical installation:
- Add the Prometheus community Helm repository:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts - Update the repo:
helm repo update - Create a namespace:
kubectl create namespace monitoring - Install the chart:
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring - Wait for pods to start:
kubectl get pods -n monitoring - Access the Prometheus UI by port-forwarding:
kubectl port-forward -n monitoring svc/prometheus-operated 9090(the service name may vary; check withkubectl get svc -n monitoring)
After installation, you’ll have Prometheus scraping the Kubernetes API, node-exporter for host metrics, and kube-state-metrics for object-level metrics. The default configuration includes alerting rules for common issues like pod restarts and high CPU usage.
Common Problems and Misconceptions
One common misconception is that Prometheus automatically monitors every application in your cluster. In reality, it only scrapes targets that expose metrics in the Prometheus format and are either discovered via annotations or explicitly configured. For custom applications, you need to add a prometheus.io/scrape: "true" annotation to the pod or create a ServiceMonitor resource.
Another frequent issue is storage: Prometheus’s default local storage is ephemeral. If the Prometheus pod restarts, data is lost. For production, you should configure persistent volumes or use remote storage like Thanos or Cortex. Also, high-cardinality metrics (e.g., unique labels per request) can cause memory issues; you may need to tune scrape intervals or use recording rules.
Finally, service discovery can fail if the Prometheus service account lacks the necessary RBAC permissions. The Helm chart creates a service account with proper roles, but if you write your own manifests, you must define ClusterRole and ClusterRoleBinding for get, list, and watch on pods, services, endpoints, and nodes.
Comparison with Alternatives
Prometheus is not the only monitoring solution for Kubernetes. Alternatives include:
- Grafana Cloud / Grafana Agent: A hosted service that uses Prometheus-compatible metrics but offloads storage and management. Good for teams that don’t want to operate their own monitoring stack.
- Thanos: Not a replacement but an extension that adds long-term storage and high availability to Prometheus. Often used alongside Prometheus for multi-cluster setups.
- VictoriaMetrics: A drop-in replacement for Prometheus with better performance and lower resource usage. It supports the same query language and service discovery.
- Datadog, New Relic, etc.: Commercial APM tools that provide more features (traces, logs) but are paid and may have a different data model.
Prometheus’s main advantages are its open-source nature, strong Kubernetes integration, and the huge ecosystem of exporters and dashboards. Its main drawbacks are the lack of native long-term storage and the need to manage scaling yourself.
Cost and Resource Considerations
Running Prometheus on Kubernetes has minimal direct cost (it’s open-source), but it consumes cluster resources. A typical small setup with kube-prometheus-stack uses about 1-2 CPU cores and 2-4 GB of RAM, depending on cluster size and scrape frequency. For large clusters, you may need to scale Prometheus horizontally (using Thanos or a sharding setup), which increases resource usage. Additionally, if you use cloud-managed Kubernetes, the persistent volumes for Prometheus storage will incur costs. To optimize, you can reduce scrape frequency, limit metrics retention, and use recording rules to pre-aggregate expensive queries.
Security and Safety Best Practices
Monitoring tools can expose sensitive information. Ensure that the Prometheus UI and API are not publicly accessible; use network policies, authentication (e.g., via an Ingress with basic auth or OAuth), or keep them behind a VPN. The default Helm chart creates a ClusterIP service, which is only reachable inside the cluster, so you’re safe unless you expose it. Also, be careful with the metrics you collect: avoid scraping secrets or sensitive data. For Alertmanager, if you configure webhooks, ensure they are over HTTPS. Finally, keep Prometheus and its components updated to patch security vulnerabilities.
Frequently Asked Questions
How do I scrape metrics from my own application?
Your application must expose an HTTP endpoint (e.g., /metrics) that returns metrics in the Prometheus text format. Then, add the annotation prometheus.io/scrape: "true" to your pod spec, and optionally prometheus.io/path and prometheus.io/port. If you use the Prometheus Operator, you can create a ServiceMonitor resource that selects your service.
What is the difference between Prometheus and Grafana?
Prometheus is a monitoring system that collects and stores metrics, while Grafana is a visualization tool that creates dashboards from various data sources, including Prometheus. They are often used together: Prometheus handles data, Grafana displays it.
How do I persist Prometheus data across restarts?
Configure a PersistentVolumeClaim for the Prometheus server’s /prometheus directory. In the Helm chart, you can set prometheus.prometheusSpec.storageSpec to define a volume claim template. For long-term storage, consider using Thanos or a remote write endpoint.
Can I monitor multiple Kubernetes clusters with one Prometheus?
Yes, but you need to configure federation or use a remote write approach. The simplest way is to run a Prometheus in each cluster and use a central Thanos or Cortex to aggregate data. Alternatively, you can point one Prometheus to multiple clusters by defining additional scrape configs with different API servers and bearer tokens.
Why are my alerts not firing?
Check that the alerting rules are correctly defined and that the metric names match. Also, verify that Alertmanager is running and configured with a receiver (e.g., email). Use the Prometheus UI’s “Alerts” page to see the state of each rule. Common issues include incorrect label matchers or missing metrics because the target isn’t scraped.
Related
- How Long Does Persephone Stay with Hades?
- Odysseus Last Name? Myth & Name Origins
- How Long Was Odysseus on Calypso’s Island?
How pages are sourced and corrected: Sources and editorial policy.