Hydrolix 5: Increased Operability, More Hyperscale

For the majority of enterprises, the volume of log and event data keeps going up, leading to major challenges when it comes to ingesting, analyzing, and retaining all that data. Hydrolix is designed for cost-effective real-time and historical analytics at petabyte scale.

Lately, we have been on a journey to hyperscale, with customers ingesting tens of millions of logs per second during major events. And because Hydrolix is built to maximize the performance of object storage, enterprises can keep all their data “hot” for queries long-term.

Hyperscale is just one part of the equation, though. As data gravity increases, it’s just as important to make data available to teams when and where they need it. That includes not just operations teams but also teams that handle business intelligence, data science, and training machine learning models.

In recent months, we’ve released connectors for Splunk, Apache Spark, and Kibana. Whether your teams are used to working with Splunk, Databricks, or Elasticsearch, we have you covered. You can dramatically reduce the total cost of ownership (TCO) of your existing solutions by keeping your log data in Hydrolix. Meanwhile, we remain committed to increasing interoperability between Hydrolix and the tools your teams use.

With Hydrolix 5.0, we remain committed to the journey. You’ve probably heard the saying good things come in threes… well, good things are also coming in Hydrolix 5. Let’s take a look at some of our exciting new features for 5.0 and beyond, including:

• Built-in Kibana support
• MySQL compatibility
• Shadow tables
• Column aliasing and renaming
• ClickHouse update
• Merge controller v2 in beta

Let’s start with the features that make data more available (and flexible) for your teams.

See the release notes to learn more about Hydrolix 5.0.

Built-In Kibana Support

Last year, we partnered with Quesma to release an integration that allows users to analyze their Hydrolix data in Kibana. Many analysts love Kibana, but traditionally, it has only been available as part of the ELK stack (Elasticsearch, Logstash, and Kibana). While Elasticsearch is a powerful tool, many enterprises are running into skyrocketing costs as their data volumes go up. Enterprise accounts are expensive, and many aspects of Elasticsearch’s architecture—including the need for replicas and tightly-coupled compute and storage—aren’t a good fit for keeping log data affordable at hyperscale.

With Hydrolix 5.0, the integration is now built into Hydrolix clusters—no setup is necessary, and you can start using Kibana right away for data analysis.

MySQL Compatibility

Many teams prefer working with MySQL, and MySQL client applications can now connect to and query Hydrolix clusters. Even better, this makes it easier to integrate Hydrolix BI tools like Looker and Tableau, allowing teams to get more insights out of their data using the tools they prefer.

Shadow Tables

It’s time to bring your testing out of the shadows with shadow tables! Building data pipelines can be extremely complex. Hydrolix simplifies the process of ingesting, transforming, compressing, indexing, and standardizing data with a built-in ingest pipeline that can autoscale to tens of millions of log lines per second—and then scale down when demand subsides so there’s no wasted compute.

What happens if your teams want to update the ingest pipeline, though? Perhaps new attributes have been added to a data source or you want to add new SQL transforms for a specific use case. Whatever the reason, data pipelines can become brittle if they aren’t updated. However, taking a pipeline offline means interruptions to your data, and testing in prod, especially on pipelines ingesting terabytes of data daily, is enough to make engineers break into a cold sweat. But there’s also no way to easily test the flow of big data in most development environments.

With shadow tables, data engineers and other technical practitioners can actively test ingest pipelines in production without interrupting the flow of incoming data. A shadow table mirrors source data into a sample table where teams can test updated transforms. This sample table is a subset of the product data. You can ensure that the data in the sample table looks how it should, make adjustments as needed, and compare the sample table to the original table. There’s no need to worry about malformed, dropped, or otherwise incomplete data—you get all the advantages of testing in prod without any of the risks. Then, when you’re satisfied with the shape of the incoming data in the shadow table, you can update the transform to the source table and allow the shadow table to return to the shadows from whence it came.

Column Aliasing and Renaming

Flexible schemas are all the rage these days. Teams need the ability to change the shape of their data, and different stakeholders might want to name and standardize data in different ways based on their specific use case. Hydrolix already includes flexible schema—you can add or remove columns as needed (such as when a data source changes), all without any impact to your existing data.

However, Hydrolix is also append-only and immutable—which is important for security and ensuring that event data remains a source of truth. Hydrolix’s append-only, immutable nature also makes it much faster—no need to worry about updates or deletes impacting incoming flows of data at hyperscale.

Renaming columns is a nice perk of flexible schema, but this feature isn’t normally compatible with immutable data sources. With Hydrolix’s renaming and aliasing features (coming soon), you can rename columns and add aliases as needed. Who says it’s not possible to be immutable and flexible at the same time?

ClickHouse Update

Our query components use ClickHouse, and we’re updating our fork of ClickHouse to 24.8.6.70, giving users more functions to work with. Teams working with big data and OLAP (online analytical processing) solutions know that aggregating data (and aggregate tables) are a powerful method for quickly and efficiently generating valuable metrics on the fly. In the case of Hydrolix, summary tables use intermediate states to handle late-arriving data and generate real-time aggregations—even on tens of millions of log lines per second.

With the updated version of ClickHouse, you can now use aggregate functions like approx_top_k, topKWeighted, and approx_top_sum. And there are also new encoding and decoding functions available, performance improvements, and lots more goodies that our engineers and teams will be synergizing with our core product.

Merge Controller V2 in Beta

Much of the work we do at Hydrolix happens deep under the hood. We focus on souping up and tuning the engine for true hyperscale, then integrate the engine with any dashboard or tool that teams prefer, giving you the best of both worlds.

Hydrolix isn’t just one engine—it consists of fully decoupled, stateless architecture that consumes data as fast as a kaiju snacking on skyscrapers. Every part of the system is designed to autoscale up (or down) independently to meet the demands of data at hyperscale—all while minimizing the amount of compute needed. And the whole system uses decoupled, cost-effective object storage so you get hyperscale and affordability.

One of these subsystems is the merge service. At ingest time, Hydrolix’s ingest service creates small partitions in parallel, allowing the system to ingest millions of lines per second. The merge service combines these partitions over time (using an immutable process), leading to greater compaction and more efficient queries. Like other systems in Hydrolix, the merge service has been tempered in the forge of massive real-time events—one recent event generated 200 terabytes of data over the course of a few hours. This provided our intake team with the insights needed to turbocharge the service. The improved performance is due to a completely redesigned merge service, which includes removal of an intermediate queue in favor of a direct RPC connection between components, more efficient in-memory algorithms, and a new query pattern that better utilizes the catalog (which contains basic information about partitions in a cluster). We’ll be writing more about the merge service in a future post.

Learn how the merge service improves compression.

Good Things Come in 5—and Beyond

We’re proud of Hydrolix 5.0—but this is really just the tip of the iceberg. We have a lot of new features and integrations that we’re excited to share in upcoming posts.

That means a dedication to creating a platform that sings at hyperscale—and that can dramatically reduce the TCO of use cases like observability, cybersecurity, and media delivery for enterprises ingesting a terabyte or more of log and event data per day.

And it also means a steady flow of new integrations designed to help teams visualize and analyze data wherever they need it. We’ll bring the engine, and you can BYOD—bring your own dashboard—whether that’s Grafana, Kibana, Splunk, or Databricks. Hydrolix is committed to maximizing interoperability so your teams can analyze data using a wide range of tools. If you’re working with massive volumes of log data, Hydrolix is the place to keep it—but the possibilities of what you can do with that data are endless.

Next Steps

• Read the release notes to learn more about Hydrolix 5.0 
• If you’re not using Hydrolix yet and would like to learn more, contact Hydrolix about a managed trial or demo.