Hydrolix or Elastic Cloud Hosted for Log Data? A Deep Dive

Elastic Cloud and the open-source ELK stack (Elasticsearch—Logstash—Kibana) are powerful tools for many use cases, but they aren’t the best fit for high volumes of log data. The reason is simple: the high cost of ingesting, storing, and analyzing daily volumes of a terabyte or more for use cases like real-time analytics.

In previous posts, we’ve compared the price, data retention windows, and ease of operations of Hydrolix versus Elastic Cloud Hosted. For log volumes of a terabyte per day or more (a use case that is increasingly common for many enterprises), Hydrolix is a better fit than Elastic Cloud Hosted. This post doesn’t cover Elastic Cloud Serverless.

In this post, we’ll do a deeper dive into each platform to show why Hydrolix is the better choice for large volumes of log data.

Hydrolix is compatible with Kibana. If you’re using the ELK stack, you no longer have to worry about vendor lock-in if you want to continue using Kibana. You can now use Kibana and KQL to analyze all your Hydrolix data.

General Purpose vs Specialization

TL;DR: Elasticsearch is first and foremost a document search engine that has since been updated to handle many other use cases. In contrast, Hydrolix is designed specifically for high volumes of log data.

Elasticsearch was originally designed as a document search engine, and that still remains its strongest use case. Elasticsearch and Elastic Cloud expanded into other use cases like real-time analytics based on the theory that its search principles could be applied to anything that relies on search, including observability and real-time analytics.

Architectural decisions that make it the right choice for document search make it less ideal for log data. For example, Elasticsearch uses a document oriented database to optimize for text search. However, OLAP-style columnar storage (like what Hydrolix uses) is better for compressing log data and providing performant searches. And while you can sometimes benefit from using text search for log data, looking for word matches in error messages is generally not the most efficient way to search your logs.

Conversely, Hydrolix is designed specifically for high volumes of log data. Every design decision in Hydrolix is made with this use case in mind.

These decisions go far beyond using columnar storage. For example, Hydrolix partitions data by time in order to optimize for time-filtered queries. And Hydrolix’s high-density compression has rates of 90% or more, making distributed object storage possible for real-time analytics. This is why Hydrolix has the lowest per GB cost for log data in the industry while keeping all data hot for low-latency queries—a necessity for getting deep insights into your log data.

Expensive SSDs Versus Commodity Hardware

TL;DR: Elasticsearch relies on expensive hardware for ingest and query performance while Hydrolix is designed to maximize the performance of cost-effective commodity hardware.

To achieve high levels of performance, Elasticsearch’s “hot” tier relies on solid state drives (SSDs), which are much more expensive than commodity hardware. Combined with the need for replicas (copies of data), the hot tier is extremely expensive for large volumes of log data.

It isn’t until you reach the cold and frozen tiers of Elasticsearch that pricing begins to get more cost-effective, but this data is difficult to access unless you have searchable snapshots, which are only available with an Elastic Cloud Enterprise account.

As the following infographic shows, Elastic Cloud is too expensive compared to Hydrolix whether you use searchable snapshots or simply throw that data away. The culprit is the high cost of hot storage at one terabyte per day. These costs only get more prohibitive at higher volumes of data.

Elasticsearch introduced lifecycle management in 2020 to offset these high costs. In addition to requiring more operations, it doesn’t truly fix the problem for high volumes of log data. While lifecycle management can make Elasticsearch more cost-effective for use cases of one TB/day or more, it’s still extremely expensive compared to Hydrolix.

In contrast, all data in Hydrolix is hot while running on commodity hardware. This is because Hydrolix is designed to maximize the performance of S3-compatible object storage, giving you sub-second query performance regardless of whether your data is a minute or a year old.

Replicas and Availability Zones Bloat Costs for Elastic Cloud

TL;DR: Elastic Cloud ensures data resiliency and high availability through replicas while ensuring application availability through availability zones. Both replicas and availability zones bloat your storage footprint and costs. Because Hydrolix uses S3-compatible storage, it can rely on cloud provider availability guarantees for both data and application availability—no replicas or availability zones required.

To ensure high availability and resiliency in Elasticsearch for hot and warm tiers, you need to have at least one replica per shard. A zero replica strategy is not recommended for production data and creates a high risk of data loss. So at the very least, you need to double the size of your storage footprint. This is a huge problem if you’re already ingesting huge volumes of data. By doubling that size, you’re drastically increasing costs and the need to scale up your cluster.

If you’re an Elastic Cloud user, you also need to have additional availability zones for production (at least two total) and mission-critical data (three total). Unfortunately, Elasticsearch requires extremely fast networks to manage replicas and doesn’t work as effectively across availability zones that use distributed networks. This means creating duplicates of your entire cluster, bloating costs further.

In contrast, Hydrolix uses S3-compatible cloud storage for all data and cloud-native Kubernetes infrastructure for components. Cloud providers have guarantees for high availability and resiliency, so there is no need for either replicas or availability zones with Hydrolix.

Making Object Storage Hot

TL;DR: Elasticsearch was designed first and foremost for performance, making it challenging to achieve cost-effectiveness later. With Hydrolix, which is focused on making S3-compatible storage performant, cost-effectiveness has been embedded in its design from the very beginning.

What is “hot” data when it comes to real-time analytics? True real-time (at the order of nanoseconds) is neither cost-effective nor necessary to act on your data quickly. As Gartner defines real-time analytics, “Real-time analytics is the discipline that applies logic and mathematics to data to provide insights for making better decisions quickly. For some use cases, real time simply means the analytics is completed within a few seconds or minutes after the arrival of new data.”

Because Elasticsearch runs its hot tier on closely-coupled, high-performance storage, it can achieve millisecond latency for use cases that need it—with the tradeoff that it can be very costly, especially at scale.

Elasticsearch was designed to be very performant, and the search for cost savings only came later as data volumes continued to grow. Features like lifecycle management are essentially bandaids. They don’t change the underlying architecture, but rather introduce new compromises where some cost savings are achieved at the expense of performance and less-accessible data. High-performance SSDs simply aren’t going to be cost-effective at scale.

Hydrolix took the opposite approach. Instead of trying to make performant storage cost-effective, Hydrolix’s co-founders started with the premise that cost-effectiveness is a basic requirement. That meant maximizing the performance of object storage for real-time analytics.

Hydrolix has focused on the strengths of object storage such as parallelism, high availability, and the ability to request narrow ranges of data. And it has responded to the challenges of distributed storage (specifically, the distance data has to travel) by minimizing the amount of data that needs to be transferred by focusing on high-density compression and optimizations like partition pruning and predicate pushdown.

The result is a solution that provides a target time-to-glass of ten seconds or less—even for massive volumes of data. This includes more than ten million log lines per second for the biggest game of the year.

Elasticsearch’s hot performance is an example of how it’s a great generalist tool, but not the best choice for specializing in high volumes of log data. Big data use cases need to be cost-effective or the data ends up discarded or unused. And even with Elasticsearch’s closely-coupled hot tier, there is plenty of potential for things to go wrong, from hotspotting to resource contention, especially at higher volumes of data.

Meanwhile, Hydrolix gives you the real time analytics performance you need at a fraction of the cost of other platforms.

Compression: Every Bit Counts

TL;DR: Elasticsearch uses general compression algorithms (LZ4 and deflate) which typically have 2x-3x compression. Hydrolix uses specialized compression algorithms and novel techniques to achieve 90%+ compression. Even Elasticsearch’s recently announced Logsdb Index Mode only achieves 3x compression.

Both Elasticsearch and Hydrolix use data compression. But every little bit counts, especially for large volumes of data. Elasticsearch uses LZ4 by default with the option to use deflate. LZ4’s default setting achieves about 2x compression while deflate achieves about 3x compression, but with additional performance tradeoffs to compress and decompress data. These are general and well-established compression algorithms that have been used for many years. Elasticsearch just announced a new Logsdb Index Mode, which is supposed to achieve 3x compression.

Hydrolix’s proprietary high-density compression achieves much higher compression rates: typically 10x or more. You also don’t need to manage or worry about compression like you do with Elasticsearch—it’s all managed for you. While much of the implementation can’t be discussed, one critical difference is that each data type has its own specialized algorithm. This is in contrast to applying a general compression algorithm to all data.

In Hydrolix, compression is closely entwined with indexing, and indexing data actually has the beneficial side effect of increasing compression further. Achieving very high compression rates is critical for making object storage performant for real-time analytics. It also leads to much greater savings when it comes to storing data. If you are ingesting a TB/day of raw data, at the end of a year, at 10x compression, the total volume of data compressed in Hydrolix will only be around 40 TB compared to around 120-180 TB with Elasticsearch with LZ4 or deflate. Even the new Logsdb Index Mode will still have approximately 120 TB in storage.

Lifecycle Management Versus an Automated Merge Service

TL;DR: Elasticsearch’s data lifecycle requires additional management and compromises the availability of your data. Hydrolix’s data lifecycle is hands-off and includes a merge service that automatically optimizes and compresses your data further over time.

Elasticsearch has a data lifecycle you have to manage while Hydrolix includes an automated merge service that runs in the background.

Elasticsearch’s data lifecycle compromises performance in exchange for increased cost-effectiveness. While the most recent data is typically more likely to be queried, there are many use cases for querying less recent data, including threat hunting, machine learning training runs, and BI and data science use cases. So how do you manage the cutoff? When should your data be considered less useful for queries?

While many enterprises come up with data lifecycle policies that are acceptable for their use cases, they reflect the underlying limitations and costs of Elasticsearch’s architecture and not necessarily the best interests of the enterprise.

Inevitably, lifecycle policies will change if original estimates are improperly scoped (common) or your log data volumes continue to grow (pretty much guaranteed). This requires additional management and careful consideration of costs versus performance and retention—with cost-cutting typically given priority because of the high costs of data in Elasticsearch.

Because all data is hot in Hydrolix, making lifecycle management much simpler. Typically, the main thing you have to consider is whether data should be removed after the default of 15 months or whether to shorten or extend the retention period. Enterprise needs are the determining factor for retention length, not architectural shortcomings. And because all data is stored in a storage bucket in your own VPC, you have full control.

Hydrolix also includes an automated merge service that optimizes storage over time. Partitions are compacted, improving compression and query performance. And because each subsystem operates independently using cloud-native Kubernetes infrastructure, there is no resource contention. Contrast that with Elasticsearch’s lifecycle management, which can use intensive resources and lead to resource contention when data is moved between tiers.

More Efficient Query Infrastructure

TL;DR: Elasticsearch queries aren’t always efficiently spread across nodes, and there can even be issues with hotspotting. Hydrolix query peers are distributed evenly between partitions during a search.

In Elasticsearch, shards hold data and are responsible for indexing and search operations. Elasticsearch servers, usually called nodes, typically hold multiple shards, and you can horizontally scale your Elasticsearch cluster by adding more nodes. Storage and compute are tightly coupled in shards, and separating them simply isn’t possible based on Elastichsearch’s architecture.

In theory, queries will be distributed across multiple nodes, but in practice, the query results may be concentrated on fewer nodes. This can lead to hotspotting where nodes are overutilized. And it shows that Elasticsearch doesn’t always fully utilize parallelism. For example, one query may be distributed across multiple nodes while the next may be concentrated in a single node. Even when this doesn’t lead to bottlenecks or issues, this inefficiency adds up over time. In addition to wasted compute, you may need to overprovision to minimize the issue.

In contrast, Hydrolix query infrastructure consists of peers that evenly handle the workload of individual queries. When a user makes a time-filtered query in Hydrolix, the query head first consults the catalog to determine how many partitions match the time range. Then the work of checking the partitions is evenly distributed among existing query peers. There’s no risk of issues like hotspotting. This is only possible because compute and storage are entirely separate.

Hydrolix is able to maximize the benefits of parallelism because each component is decoupled and stateless. This simply wouldn’t be possible with Elasticsearch’s design.

Scaling Up In Response to Demand

TL;DR: Only Hydrolix can autoscale quickly when demand goes up. With Elasticsearch, adding nodes is a resource-intensive process that triggers shard rebalancing.

Both Elasticsearch and Hydrolix have the ability to scale up over time. However, Hydrolix includes autoscaling that can adjust resources on the fly based on demand. This simply isn’t possible with Elasticsearch.

Why is this so important? Many enterprises can have major variations in both log intake and query compute. For example, media and streaming providers need to scale up for live sporting events and retail companies need to scale for major sales. And regardless of vertical, major incidents, DDoS attacks, and other surprises can also trigger huge spikes in logs.

To scale Elasticsearch, you need to add new nodes, which is an expensive operation that triggers shard rebalancing. Because storage, ingest, and query are tightly coupled with Elasticsearch, adding new nodes impacts performance across both ingest and query.

In other words, if you need to autoscale quickly, such as to deal with a major incident or unexpected traffic, you’ll instead be dealing with a resource bottleneck while shards are rebalancing.

In contrast, Hydrolix’s stateless Kubernetes architecture makes it easy to scale each part of the system independently. Systems autoscale to meet demand, and you also have the option to manually scale ahead of planned major events. And unlike Elasticsearch, Hydrolix’s scaling process doesn’t trigger compute-intensive processes like rebalancing.

It’s Not Just about Scaling Up—What About Scaling Down?

TL;DR: Hydrolix can also easily scale down to prevent overprovisioning, while scaling down in Elasticsearch is a painstaking process.

Elasticsearch’s issue with scaling goes the other direction, too. Only Hydrolix has the ability to scale down gracefully. With Elasticsearch, scaling down is a painstaking process that can lead to cascading failures.

If you’re dealing with spikes in your log data, once an event or incident is over and demand subsides, you’ll be overprovisioning if you don’t scale down, leading to high costs.

If you can’t scale down, you either need to provision for the highest possible load, add a streaming mechanism that buffers data, or downsample during major spikes. None of these options are ideal—the first is expensive and wasteful, the second leads to reduced performance when you most need data, and the last results in lower data resolution and data gaps.

To scale down with Elasticsearch, you need to remove nodes. First, the node must be drained, with all data from the node transferred to remaining nodes. This process is resource-intensive and comes with many gotchas. Just as the node is being taken down, leading to fewer resources in the system, there’s also a spike in research usage due to data transfer. If anything goes wrong, it can lead to cascading failures that negatively impact your cluster. For instance, as the process takes more resources, it can cause other nodes to have issues with consumption and even go down.

In contrast, Hydrolix can autoscale up or down in response to traffic. It’s easy to gracefully scale down after a major event, and you can even scale down to zero resources if needed.

Dealing With Resource Contention

TL;DR: Elasticsearch’s tightly-coupled architecture can lead to issues with resource contention. With Hydrolix, you can easily set up query pools and separate workloads to prevent resource contention.

With Elasticsearch, resource contention can be a major issue for teams, especially during periods of high demand. Since shards handle indexing, querying, and storage, the ingest process can interfere with querying, especially during periods with high volumes of read and write operations. For instance, during major events, teams may be ingesting millions of lines per second while using many dashboards and querying that data as it’s ingested. And if there’s an incident, the demand for query compute will go up further.

As a result, you can have stakeholders competing for the same compute resources while nodes are also contributing to resource contention due to the fact they are tightly coupled. This can lead to significant performance issues in Elasticsearch when you most need access to your data.

While you can use nodes that are specialized just for ingest to reduce the problem of resource contention between ingest and query, adding more nodes to the system will increase your costs and the need for maintenance further.

Some teams may have a higher demand for compute resources than others, creating what’s known as the noisy neighbor effect. A noisy neighbor could be a team running many large, compute-intensive queries or it could even be an automated process like a machine learning training run.

There is only one way to make workloads independent and eliminate resource contention in Elasticsearch. You have to make a separate cluster for each workload, which will bloat storage and costs while leading to additional maintenance overhead. This level of denormalization isn’t just extremely expensive—it will create a lot of headaches when it comes to keeping data consistent across clusters.

Not surprisingly, this isn’t feasible for many teams. So if an enterprise wants to do machine learning training runs for use cases like anomaly detection, they have to deal with resource contention or perhaps set up jobs during off-peak hours, limiting flexibility.

In contrast, you can set up separate workloads for query and ingest in Hydrolix. In the case of query, you can use precision query scaling to finetune separate workloads for each team. That includes the number of query peers that a workload uses (higher means more parallelism) as well as memory and CPU. In addition to eliminating the possibility of resource contention, it allows you to balance cost versus performance for each team and workload depending on how mission-critical it is.

Estimating Costs Versus Pay as You Go (PAYG)

TL;DR: With Elastic Cloud, estimating costs and prepaying for credits can lead to surprise bills and high costs as log volumes go up or estimates are miscalculated when contracts are signed. Hydrolix uses a straightforward license structure and you use your own cloud compute.

Surprise bills are all too common for Elastic Cloud. Too often, teams burn through a year’s worth of credits well before the year is up—and may even burn through them in six months. There are multiple reasons this issue happens:

• It’s challenging to predict how many resources an Elastic Cloud implementation will use, and enterprises often get billing estimates that are considerably lower than the actual cost.
• Many enterprises are seeing massive increases in log volumes while data teams must handle new KPIs and requests from internal stakeholders.

The result can be major sticker shock—and a $2 million dollar estimated spend for the year quickly doubling or tripling. Too often, teams must make additional compromises on data collection and retention in order to keep costs down, leading to more management and potential friction with the teams that rely on incoming log data.

In contrast, Hydrolix uses a simple licensing model and you pay for your own compute and storage through your cloud provider. In addition to being able to negotiate volume discounts with your cloud provider, you get full cost transparency through your provider and Hydrolix adds no additional costs to compute or storage usage.

While you’ll still have to navigate increases in log data volumes, you’ll be able to do so incrementally, automatically, and without overprovisioning.

And because Hydrolix is so much more cost-effective than Elastic Cloud, you’ll have more flexibility to absorb any surprise increases in your log data volume, retain data as long as you need, and still dramatically reduce your total cost of ownership (TOC).

Fully-Managed Versus Partially-Managed

TL;DR: Even with enterprise-level support, your Elastic Cloud service will not be fully-managed. Hydrolix is a managed service that’s much simpler to operate.

Elastic Cloud customers who choose the enterprise tier often do so to get enterprise-level support. And yet even with an enterprise account on Elastic Cloud, you will still need to manage and operate your own ELK stack. This can be extraordinarily complex for large volumes of data. It’s typical for enterprises to need to pay extra for Elastic’s professional services, or to go through one of many boutique consulting firms that can help manage an Elasticsearch cluster.

The risks of going it alone are high—a misconfigured cluster can lead to major issues with substandard performance and unexpected costs. And yet professional services also add significant, often unanticipated costs that aren’t part of the original contract.

Meanwhile, Hydrolix offers a fully-managed solution. Hydrolix manages the deployment, initial configuration, and Kubernetes infrastructure. Hydrolix also supports teams on creating aggregate tables as needed, and then users can define their queries and data visualizations.

Both Elastic Cloud and Hydrolix offer multiple tiers of support and professional services as needed.

And like Elastic Cloud, Hydrolix also offers enterprises the option to fully self-manage their Hydrolix infrastructure, providing complete control.

Regardless of whether enterprises choose self-managed or fully-managed for Hydrolix, all infrastructure and storage exists in the customer’s cloud infrastructure. This can be either through a cloud provider such as AWS or an on-prem private cloud.

Hydrolix: A Truly Cloud-Native Solution

TL;DR: Hydrolix is a true cloud-native solution that maximizes the benefit of the cloud. Elastic Cloud adds many cloud features but isn’t built on a cloud-native solution.

Unlike Elasticsearch or Elastic Cloud, Hydrolix is a true cloud-native solution. Elastic Cloud includes implementations like Elasticsearch on Kubernetes, but it relies on Elasticsearch’s tightly-coupled architecture.

Cloud-native solutions use automation, stateless components, and cloud-based components like Kubernetes and cloud storage to maximize the benefits of the cloud while reducing costs. This is the case with Hydrolix, which uses autoscaling and stateless subsystems while achieving high-performance search and ingest alongside cost-effective object storage. Hydrolix is horizontally scalable and relies on the cloud’s inherent guarantees of resiliency and high availability.

Elastic Cloud’s approach does not rebuild Elasticsearch from the ground up for cloud. And this is reminiscent of the challenge that many organizations have with cloud migrations. It is not atypical for enterprises to move from on-prem to the cloud, only to find that the benefits they’re expecting, such as reduced costs, never materialize. The issue isn’t the cloud, but the fact that the stack isn’t adapted to maximize the benefits of cloud. To actually maximize those benefits, enterprises must take a cloud-native approach.

Elasticsearch’s reliance on replicas and stateful, tightly-coupled components can lead to high costs and scalability issues, especially for enterprises ingesting terabyte-volumes of log data every day. And when it comes to Elastic Cloud, you’ll get a platform that operates in the cloud but simply doesn’t have all the benefits of being a cloud-native solution.

When Should You Use Hydrolix Instead of Elasticsearch?

TL;DR: Choose Hydrolix for ingesting, storing, and analyzing at least one terabyte of log data per day. For use cases other than log data, it’s best to use Elasticsearch or another solution.

Elasticsearch is a well-loved solution, and for good reason. It is amazing for document search, and it can be effective for many other search use cases as well. When it comes to log data, solutions like Elastic Cloud can handle lower volumes of log data and provide a well-developed UI for features like APM and observability.

However, Hydrolix excels for log data use cases of at least one terabyte per day. You can use Hydrolix as a standalone solution for your log data or to complement your ELK stack implementation. Either way, you can analyze all of your Hydrolix log data in Kibana, including alongside your Elasticsearch data stores.

Next Steps

• Check out our infographic comparing the cost of Hydrolix and Elastic Cloud.
• Learn more about how Hydroix compares to Elastic Cloud when it comes to cost, data retention, and ease of operations.
• Ready to try out Hydrolix? Contact Hydrolix about a proof of concept or demo.