Visualizing OpenTelemetry Data With Hydrolix and Grafana

OpenTelemetry excels at collecting comprehensive, high-fidelity telemetry data. But that comes with a challenge: at scale, you might generate millions of spans per minute. Many observability platforms have challenges at this scale, leading to aggressive sampling. In contrast, Hydrolix is designed to provide high-fidelity data at any scale, allowing you to add traces without compromises.

By combining OpenTelemetry’s collection capabilities with the storage and query performance of Hydrolix, you get complete visibility without the traditional cost penalty. You don’t have to aggressively sample your traces or delete data after a few days, and you can keep all your data and query it quickly.

This post will cover the challenges of modern observability with OpenTelemetry, how to combine Hydrolix and OTel, and how to visualize traces with Hydrolix and Grafana.

The Challenge of Modern Observability With OpenTelemetry

Modern applications are more complex than ever. Microservices architectures, distributed systems, and cloud-native deployments have revolutionized how we build software, but they’ve also made understanding system behavior much harder. When performance degrades or errors spike, you need to know about it immediately, before your users notice, support tickets pile up, and revenue is impacted.

If you’re learning about problems from user reports, you’re already too late. Your observability strategy has failed.

OpenTelemetry changes this equation. With comprehensive instrumentation across your entire stack, you can detect anomalies in real-time, trace the exact path of failing requests, and identify performance degradation the moment it begins. You shift from reactive firefighting to proactive problem resolution. However, implementing effective observability at scale generates massive volumes of high-cardinality data. Traditional databases buckle under the weight of millions of traces per minute, forcing teams to make painful tradeoffs—aggressive sampling that makes it harder to identify issues, retention windows measured in days instead of months, and huge storage costs that eat up your budget.

What if you didn’t have to choose between comprehensive observability and cost-effectiveness? With Hydrolix, which is designed for petabyte-scale data, you don’t have to sacrifice one for the other. And now, with the the Hydrolix OpenTelemetry collector, you have more ways to ingest high-volume telemetry data into Hydrolix.

Hydrolix is designed for high-volume, full-fidelity data. Many platforms force you to choose between high costs and keeping your data. Hydrolix allows you to retain 100% of your data long-term for real-time queries—all while paying a fraction of the cost of traditional observability platforms.

Book a demo with Hydrolix.

Combining OTel and Hydrolix

OTel works by installing a language SDK on your application. The SDK sends data to an endpoint or an OTel collector. Hydrolix has a custom collector image with an embedded Hydrolix exporter that you can use to send data to Hydrolix. You can then view data in the visualization tool of your choice. The next image shows this process, with Grafana as the visualization tool.

Here’s a walkthrough of how to set up OTel with Hydrolix.

Install a Language SDK

OpenTelemetry provides native SDKs for virtually every major programming language and framework, including Java, Python, Go, .NET, JavaScript, Ruby, PHP, and more. Each SDK includes both manual instrumentation APIs and automatic instrumentation for popular libraries and frameworks, enabling you to start collecting telemetry with minimal code changes.

Visit the OpenTelemetry documentation to find installation instructions and quick start guides specific to your stack. Here’s an example code snippet for instrumenting a Next.js frontend.

# app/layout.tsx
import React from "react";
import "../src/telemetry/feTelemetry";
export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html lang="en">
      <body>
        {children}
      </body>
    </html>
  );
}

Initialize OpenTelemetry in the src/telemetry/feTelemetry.ts file. Import this file as early as possible in your application startup, ideally before any routes, API calls, or user interactions are initialized. This ensures all actions are properly instrumented and traced. This example uses automated web instrumentation using the @opentelemetry/auto-instrumentations-web package. It sends metrics and spans to the Collector via the HTTP exporter.

Note: The HTTP exporter shown here is for demonstration purposes; the Hydrolix exporter doesn’t require it.

# src/telemetry/feTelemetry.ts
"use client";
import {
  CompositePropagator,
  W3CBaggagePropagator,
  W3CTraceContextPropagator,
} from "@opentelemetry/core";
import { WebTracerProvider } from "@opentelemetry/sdk-trace-web";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
import { getWebAutoInstrumentations } from "@opentelemetry/auto-instrumentations-web";
import { Resource } from "@opentelemetry/resources";
import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { metrics } from "@opentelemetry/api";
import {
  MeterProvider,
  PeriodicExportingMetricReader
} from "@opentelemetry/sdk-metrics";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
const initializeMetrics = () => {
  const metricExporter = new OTLPMetricExporter({
    url: process.env.NEXT_PUBLIC_OTEL_METRICS_URL,
  });
  const metricsReader = new PeriodicExportingMetricReader({
    exporter: metricExporter,
    exportIntervalMillis: 5000,
  });
  const resource = new Resource({
    [ATTR_SERVICE_NAME]: process.env.NEXT_PUBLIC_OTEL_SERVICE_NAME_FE,
  });
  const readers = [metricsReader];
  const metricProvider = new MeterProvider({
    resource: resource,
    readers: readers,
  });
  metrics.setGlobalMeterProvider(metricProvider);
};
initializeMetrics();
// Initialize TracerProvider immediately for browser (before creating spans)
if (typeof window !== "undefined") {
  import("@opentelemetry/context-zone").then(({ ZoneContextManager }) => {
    const resource = new Resource({
      [ATTR_SERVICE_NAME]: process.env.NEXT_PUBLIC_OTEL_SERVICE_NAME_FE,
    });
    const exporter = new OTLPTraceExporter({
      url: process.env.NEXT_PUBLIC_OTEL_TRACE_URL,
    });
    const provider = new WebTracerProvider({
      resource,
      spanProcessors: [
        new BatchSpanProcessor(exporter, {
          scheduledDelayMillis: 500,
        }),
      ],
    });
    const contextManager = new ZoneContextManager();
    provider.register({
      contextManager,
      propagator: new CompositePropagator({
        propagators: [
          new W3CBaggagePropagator(),
          new W3CTraceContextPropagator(),
        ],
      }),
    });
    registerInstrumentations({
      tracerProvider: provider,
      instrumentations: [
        getWebAutoInstrumentations({
          "@opentelemetry/instrumentation-fetch": {
            propagateTraceHeaderCorsUrls: /.*/,
            clearTimingResources: true,
          },
        }),
      ],
    });
  });
}

Set environment variables either by using a .env file or by exporting them directly.

# .env 
NEXT_PUBLIC_OTEL_TRACE_URL=http://localhost:4318/v1/traces
NEXT_PUBLIC_OTEL_METRICS_URL=http://localhost:4318/v1/metrics
NEXT_PUBLIC_OTEL_SERVICE_NAME_FE=local-my-app

Deploy Collector

Hydrolix has created an open-source exporter as an add-on to the official OTel collector. The collector can be built from this source or you can pull the pre-built image from the GCP’s container registry.

It’s the standard OpenTelemetry Collector you know and trust, enhanced with native Hydrolix export capabilities. Everything you’re familiar with, such as receivers, processors, batching, and retry logic, works exactly the same way. The only difference is the addition of the Hydrolix exporter, giving you a direct pipeline from your telemetry data to Hydrolix.

Configuration

OTel collectors use a YAML configuration file to manage all settings. You can use all your existing configurations and simply add the Hydrolix exporter alongside them.

The Hydrolix exporter requires a few additional settings. Here’s an example metrics exporter configuration:

exporters:
  hydrolix/metrics:
    endpoint: https://your-hydrolix-cluster.hydrolix.io/ingest/event
    hdx_table: myorg.metrics
    hdx_transform: tf_otel_metrics
    hdx_username: ${env:HYDROLIX_USERNAME}
    hdx_password: ${env:HYDROLIX_PASSWORD}

endpoint: The Hydrolix cluster ingest endpoint
hdx_table: The destination table in Hydrolix
hdx_transform: The transform that processes incoming data
hdx_username/hdx_password: Authentication credentials (use environment variables for security)

See the OpenTelemetry documentation for comprehensive collector configuration options.

See up-to-date installation instructions for the Hydrolix exporter and transforms for ingesting OpenTelemetry data.

View Traces in Grafana

Determining what caused an incident and why is essential. Tracing can provide an in-depth view of what’s happening with individual requests, error information, and bottlenecks. With tracing, you can quickly see where slowdowns and errors occur and why.

Hydrolix ships with Grafana (or your choice of visualization tool), and Grafana has a built-in traces view that visualizes OTel traces.

The Grafana trace view requires adding a query that retrieves all spans for a specific trace, with the correct column names. The following query will accept a trace ID from a Grafana variable that includes the trace name. Adding the name of the trace to the variable makes it easier to use the dropdown. The variable is populated using the concat sql command: CONCAT(name, ' (', traceId, ')').

SELECT
  traceId AS traceID,
  spanId AS spanID,
  name AS operationName,
  parentSpanId AS parentSpanID,
  serviceName AS serviceName,
  statusCode,
  dateDiff('ms', startTime, endTime) AS duration,
  startTime,
  arrayMap(key -> map('key', key, 'value',"tags"[key]), mapKeys("tags")) as tags,
  arrayMap(key -> map('key', key, 'value',"serviceTags"[key]), mapKeys("serviceTags")) as serviceTags,
  arrayMap(
    log_item -> tuple(
      log_item['name'],
      log_item['timestamp'],
      arrayMap(
        kv -> map('key', kv.1, 'value', kv.2),
        JSONExtractKeysAndValues(assumeNotNull(log_item['field']), 'String')
      )
    )::Tuple(
      name String,
      timestamp String,
      fields Array(Map(String, String))
    ),
    logs
  ) AS logs
  FROM YOUR_SPANS_TABLE
  WHERE traceId = extractAll('${traceid}', '\\((.+)\\)')[1]
    AND timestamp >= $__fromTime AND timestamp <= $__toTime
ORDER BY timestamp DESC

The following screenshot shows an example of visualizing traces stored in Hydrolix using Grafana.

Best Practices for Setting Up OTel

Regardless of whether you are using Hydrolix or another platform for traces, here are some recommended best practices you should follow to get the most out of your data.

Reduce Noise

Too much data can become overwhelming, slow down your queries, and make it harder to find the data that you care about. Filtering out unnecessary noise ensures that the right data is collected. This noise isn’t just storage waste; it causes engineers to spend more time filtering through the sea of irrelevant data.

Filter Out Health Checks

Health checks can dominate trace volume and add no real value. You can easily filter them out in the collector, reducing network costs and data volume.

processors:
  filter/healthchecks:
    traces:
      span:
        - 'attributes["http.target"] == "/health"'

High Metric Cardinality

Metric cardinality is the number of combinations of attributes that exist for each metric. It can be tempting to include high-cardinality attributes such as instance_id, job_id, and pod_name in metrics, but doing so can impact performance when you’re working with petabyte-scale data. Metrics provide an aggregated view and are indispensable for alerts. Bloating metric cardinality can slow queries and affect alerts. The right place for high-cardinality data is in spans and logs. For metrics, use low-cardinality attributes like environment, region, and method.

Focus on Actionable Alerts

Good alerting requires balance. Too many alerts cause fatigue, and engineers start ignoring notifications or waste time filtering noise from the real signals. Too few alerts mean you miss outages entirely, and your time-to-resolution (TTR) suffers.

Every alert must be actionable. Before creating an alert, define the remediation steps. If there’s no clear action to take when it fires, delete it.

Alerts that oscillate between firing and resolving, or that often fire due to false positives, are called flappy alerts. These alerts are especially destructive. They train engineers that some alerts might not be worth looking at and to ignore them. When you see flapping, investigate the root cause. Is the threshold too sensitive? Is there an intermittent issue that needs permanent fixing? Is the alert detecting a symptom rather than the actual problem?

Ensure that the teams (such as SREs and other engineers) that respond to alerts have the permissions to modify or remove them. On-call engineers have direct experience with which alerts matter and which don’t.