Datadog is a powerful observability platform that gives your teams a single place to correlate metrics, logs, traces, and alerts. There’s just one problem: it’s very expensive at scale. What are you supposed to do when you get a huge bill and you’re forced to choose between your data and your bottom line?
Sampling and short retention windows can help reduce costs, but they lead to gaps in data. What if you want to keep your data and your Datadog so you can get more value out of your analytics, all while keeping costs down?
Jonathan Mercereau, Principal Site Reliability Engineer at NVIDIA, has built and open sourced a metrics exporter to solve this problem. With this solution, you can store raw data in Hydrolix and then export metrics to Datadog and other observability solutions.
According to the repo’s ARCHITECTURE.md file, “The collector polls Hydrolix for log data, transforms it into pre-aggregated time-series metrics, and fans them out to one or more metric sinks (Datadog, Prometheus, OpenTelemetry, StatsD). Queries are defined declaratively in a YAML config file—no Go code changes are needed to add, modify, or remove queries.”
This post will cover how to combine Hydrolix with the metrics collector and Datadog so you can get more comprehensive observability without compromises on cost, data fidelity, or retention.
Combining Hydrolix and Datadog for High-Volume Data
Datadog does many things exceptionally well, including APM, correlating metrics and traces, and alerting. Teams that have built years of dashboards, monitors, and on-call workflows around Datadog will typically prefer to stay with the platform instead of migrating to other solutions.
However, as data scales up, Datadog becomes extremely costly. Datadog charges for custom metrics per unique metric series, and every distinct combination of metric name and tag values counts towards the cost. With massive data sources like CDNs, that cardinality compounds quickly, leading to huge bills if you want to retain all your data.
To solve this problem, you can route logs from high-volume sources to Hydrolix, which is designed for real-time and historical analytics on full-fidelity, petabyte-scale data. Hydrolix is built for extreme scale such as the Super Bowl, where the platform ingested 200 terabytes of data over the course of the event, with data available for queries in seconds, and median query time under half a second. Because Hydrolix combines high-density compression (10-50x) with object storage, you can retain full-fidelity data long-term (15 months by default), all with low-latency precision queries and no tiering or cold storage penalties.
The Hydrolix metrics exporter that Mercereau designed brings the two platforms together so you get the best of both worlds. As Mercereau says in his writeup of the solution design, “Raw logs answer ‘what happened?’ while time-series metrics answer ‘how are we doing?’ This tool bridges the two—it reads the raw data in Hydrolix and produces the metric signal that powers dashboards, alerts, and capacity planning.”
Real-time summary tables are a differentiating feature of Hydrolix, and they provide fresh, accurate metrics for exporting. Summary tables, which hold aggregates, are updated in real time alongside raw data tables, all while keeping the underlying raw data available for use cases like root cause analysis, forensic investigations, and compliance. This is in contrast with many platforms that provide data roll ups but then discard the underlying raw data.
Get started with CDN Insights, which gives you prebuilt summary tables with critical metrics on performance and errors across all your CDN providers.
How the Hydrolix Metrics Exporter Works
hydrolix-metrics-go is a lightweight service that polls Hydrolix at regular intervals (15 seconds by default, and fully configurable). It is a community-built tool that is now in active development. It runs SQL queries against a sliding time window and emits the results as metrics to any combination of the four supported sinks. In addition to Datadog, the solution can sink to Prometheus, OpenTelemetry, and StatsD. Each of these open source tools has a robust system of integrations, making it easier to send metrics from Hydrolix to your preferred platform.
The query window accounts for any lags in log delivery. By default, the window runs from six minutes ago to one minute ago, providing a five-minute coverage window. As Mercereau explains in the design documentation, the one-minute end offset accommodates vendor SLAs for log delivery (95% of logs delivered within 30 seconds) plus time for indexing and ETL jobs to complete. The window also enables backfill. If the collector restarts, the next poll re-covers the same window and fills any gap.
The embedded default configuration ships with Akamai CDN queries, covering edge request volume, origin traffic, error rates, and performance percentiles (p25 through p99 for first-byte time, TLS handshake, transfer, and download). These are metrics that are crucial for CDN operations teams, and can be used to create actionable alerts.
# From queries.yaml — edge requests query with dimensional tags
- name: akamai_edge_requests
sql_file: queries/akamai/edge_requests.sql
dimensions:
- column: hostname
tag: hostname
- column: status_class
tag: status_class
- column: cache_status
tag: cache_status
metrics:
- column: edge_requests_total
name: cdn.delivery.edge
unit: request
type: gaugeYou can create your own custom data sources and fields with just a SQL file and a YAML entry. No additional code configuration is required.
With this approach, the volume of custom metrics the collector sends to Datadog is determined by your query dimensions and the cardinality of your tag values, not by the underlying log volume. Even if you’re processing billions of CDN log records per day, you’ll only emit the metric series you define in your YAML configuration.
If log volume is driving up your observability costs in Datadog, ingesting and storing high-volume sources into Hydrolix instead can dramatically reduce your Datadog bill while also allowing you to keep full-fidelity data as long as you need it.
Hydrolix and Datadog, Without Compromises
If you’re using Datadog and dealing with high costs at scale, you can combine the open source metrics collector with Hydrolix to complement your existing workflows without forcing difficult tradeoffs. Keep full-fidelity data at any scale, retain it cost-effectively for as long as you need it, and continue using Datadog for the metrics, alerting, and dashboards your teams already rely on, all while cutting your observability bill. And stay tuned for further development in this area!
- Check out the open source metrics exporter and try it out for yourself.
- Interested in getting a quick value assessment on how Hydrolix can support your observability infrastructure? Get a demo.
