Prometheus long term storage is not built into the core server; the standard local TSDB is designed for short retention (often 15 days to a few months), so long term storage requires integrating an external system like Thanos, Cortex, Mimir, or VictoriaMetrics. These systems extend Prometheus’s metric retention to years while keeping the PromQL query language intact.
Why Prometheus’s Built-In Storage Is Short Term
Prometheus was designed as a monitoring system for dynamic cloud environments, not as a historical data warehouse. Its local time-series database (TSDB) uses an append-only log with periodic compaction into blocks. Each block covers two hours of data and is memory-mapped, which makes recent queries fast but makes the entire database unsuitable for multi-year retention on a single disk. The default --storage.tsdb.retention.time is 15 days, and while you can raise it, doing so on one node creates several problems:
- Disk space grows linearly with retention; a year of high-cardinality metrics can easily reach terabytes.
- Queries over long ranges become slow because the server must read many blocks from disk.
- Backups and restores become unwieldy, and a single disk failure loses all history.
- Scaling out requires sharding, which breaks global queries across the cluster.
For these reasons, the Prometheus documentation explicitly recommends using an external system for long term storage.
How Long Term Storage Integrations Work
All major long term storage solutions share a similar architecture: they ingest Prometheus’s remote write protocol (or read from object storage) and serve queries via the Prometheus HTTP API. The key components are:
- Remote write: Prometheus is configured with
remote_writeURLs pointing to the storage backend. It batches samples and sends them over HTTP, usually with snappy compression. - Object storage: Thanos and Mimir store compacted blocks in S3, GCS, or Azure Blob. This decouples storage from compute and makes data durable and cheap.
- Query frontend: The system receives PromQL queries, fetches relevant blocks, and returns results. Prometheus itself still serves short-term queries from its local TSDB.
- Compaction and downsampling: Long term systems periodically merge small blocks into larger ones and can create downsampled copies (e.g., 5-minute or 1-hour resolution) to speed up long-range queries.
Because the query API is compatible, you can point Grafana at the long term store instead of Prometheus and keep the same dashboards.
Main Options: Thanos, Mimir, Cortex, VictoriaMetrics
Four projects dominate the space, each with a different tradeoff:
- Thanos – A sidecar or receiver that uploads Prometheus blocks to object storage. It adds a Querier component that can merge data from multiple Prometheus instances and the object store. It is the most popular choice for teams that already run Prometheus and want to keep it.
- Mimir – A horizontally scalable, multi-tenant TSDB from Grafana Labs. It is a rewrite of Cortex with better performance and simpler operations. It ingests remote writes directly and stores data in object storage.
- Cortex – The predecessor to Mimir, still used in some production setups. It is more complex to operate and is largely superseded by Mimir.
- VictoriaMetrics – A single-binary TSDB that is compatible with Prometheus remote write and PromQL. It is much simpler to run than Thanos or Mimir, and it can store years of data on a single server with good compression. It also supports downsampling and retention filters per metric.
There is also Prometheus’s own “remote read” feature, which lets you query external databases, but it does not provide storage itself.
How to Choose a Long Term Storage Solution
The right choice depends on your scale, team skills, and infrastructure. Consider these factors:
- Team size: If you have one or two people, VictoriaMetrics is far easier to run than Thanos or Mimir. Thanos requires operating a sidecar, Querier, Compactor, and Store Gateway.
- Multi-tenancy: Mimir and Cortex support multiple teams with separate namespaces and auth. Thanos and VictoriaMetrics do not have built-in multi-tenancy (though you can fake it with separate instances).
- Existing Prometheus setup: Thanos works best if you already run Prometheus with local storage and want to retain that data. Mimir and VictoriaMetrics can replace Prometheus entirely for long term data, but you still need Prometheus for alerting and recording rules unless you use their built-in rule engines.
- Query performance: For very large cardinality (millions of series), Mimir and Thanos with object storage are proven at massive scale. VictoriaMetrics is also fast but may require more memory for huge workloads.
- Cost: Object storage (S3, GCS) is cheap per GB, but you pay for egress when querying. VictoriaMetrics on a large SSD can be cheaper for moderate data volumes because it avoids object storage API costs.
As a rule of thumb: if you need multi-tenancy or already use Grafana Cloud, choose Mimir. If you want to keep your existing Prometheus and add long term retention, choose Thanos. If you want the simplest possible setup, choose VictoriaMetrics.
Common Problems and Misconceptions
One common misconception is that remote write is lossy or that you must choose between Prometheus and the long term store. In reality, you can run both: Prometheus keeps its local data for fast alerting, while remote write sends a copy to the long term system. If the remote store is down, Prometheus buffers samples in a write-ahead log and retries.
Another issue is high cardinality. If you have a label like user_id with millions of values, any long term store will struggle. The solution is to reduce cardinality before sending data, for example by using recording rules to aggregate or by dropping high-cardinality labels in the remote write config.
Downsampling is also misunderstood. Thanos and Mimir downsample only after a certain time (e.g., 40 hours), so recent data is full resolution. This means long-range queries are fast but lose precision for older data, which is usually acceptable for trend analysis.
Finally, retention is not automatic in all systems. In Thanos, you must configure the Compactor’s retention period; otherwise, data accumulates indefinitely. In VictoriaMetrics, you set -retentionPeriod.
Costs and Operational Considerations
Long term storage costs are dominated by object storage and egress. As a rough example, storing 10 TB of compressed metrics in S3 Standard costs about $230 per month, plus a few dollars per million GET requests when querying. In contrast, a single large SSD (4 TB) costs around $300 once, but you must manage replication and backups yourself.
Operationally, Thanos and Mimir require more moving parts: you need to run a Compactor to merge blocks, a Store Gateway to read from object storage, and a Querier to handle requests. VictoriaMetrics is a single binary that handles everything, but it is not as battle-tested at the extreme scale of Thanos in large enterprises.
You also need to monitor the long term store itself. Use the built-in metrics endpoints (e.g., /metrics on Thanos components) and set alerts for failed uploads, high query latency, and disk fill rates.
Frequently Asked Questions
Can I use Prometheus alone for a year of data?
Technically yes, but it is not recommended. You would need a very large disk, queries would be slow, and you would have no redundancy. The official docs advise against it.
Does remote write affect Prometheus performance?
Minimal. It adds a small amount of CPU and memory for batching and compression. If the remote endpoint is slow, you can increase remote_write queue capacity to avoid dropping samples.
What is the difference between remote write and remote read?
Remote write sends data out of Prometheus to a store. Remote read lets Prometheus query an external database for data it does not have locally. They are independent; you can use one without the other.
Can I migrate my existing Prometheus data to Thanos?
Yes. Thanos sidecar uploads existing local blocks to object storage. You can also use thanos tools bucket to upload blocks from a Prometheus that is no longer running.
Is VictoriaMetrics fully compatible with PromQL?
It supports most of PromQL, but there are minor differences in edge cases (e.g., handling of rate() with counter resets). For standard dashboards, it works fine.
How long does data take to appear in the long term store?
Remote write batches data every few seconds, so it is near real-time. However, Thanos only compacts and makes blocks available for querying after a few hours, so very recent data might come from Prometheus itself, not the object store.
Related
- Where the Sausage Gets Made: The Real Answer
- How Long Did Odysseus Take to Return Home?
- Can You See Odysseus in Lake Tahoe? Answer
How pages are sourced and corrected: Sources and editorial policy.