If your enterprise is handling rising volumes of log data, observability is key. You need a tool that can ingest, store, and analyze all your log data on a daily basis while remaining performant and cost-effective.
Elastic Cloud, a managed service for Elasticsearch, is a popular choice but struggles with large volumes of log data due to its prohibitively high costs, which leads to short “hot” data retention windows. To make matters more challenging, it can be complex to operate and scale.
Hydrolix is the better fit for high volumes of log data. For enterprises ingesting at least a terabyte per day, Hydrolix is much more cost-effective than Elastic Cloud, includes long-term hot data retention, and is much simpler to operate.
This post will compare Elastic Cloud and Hydrolix on the following points:
- Cost
- Data retention
- Ease of operations
Hydrolix is compatible with Kibana. If you’re using the ELK stack, you can easily migrate to Hydrolix and keep Kibana. With our Quesma integration, you can now use Kibana and KQL to analyze all your Hydrolix data.
Comparing the Cost of Hydrolix vs Elastic Cloud
Elastic Cloud is extremely expensive for large volumes of data. In a recent post, we compared several Elastic Cloud pricing scenarios against Hydrolix. As the following infographic shows, the math just doesn’t work out for Elastic Cloud. This infographic compares Elastic Cloud versus Hydrolix for one terabyte of log data per day.
In these scenarios, Elastic Cloud is 3x–7x times more expensive than Hydrolix. And the most cost-effective scenario for Elastic Cloud discussed here involves discarding data after just 7 days—compared to long-term, “hot” data retention of a year in Hydrolix.
As the infographic shows, Elastic Cloud is already extremely expensive for enterprises ingesting one terabyte of log data per day. Those costs become even more prohibitive as log volumes continue to go up.
And most enterprises are dealing with increasing log volumes. According to a 2023 survey, log data has grown 5x over the past three years on average, and nearly a quarter of respondents (22%) experienced a growth rate of 10x or higher. Meanwhile, according to the same survey, 91% said their leadership will more closely scrutinize observability and monitoring costs.
The combination of higher log volumes and more scrutiny over costs puts teams in a difficult situation. They must reduce costs while minimizing gaps in their monitoring. For teams using Elastic Cloud, this will mean migrating some or all of their monitoring to other solutions. And it means that emphasis must be placed on evaluating and choosing solutions that are designed for high volumes of log data.
With Hydrolix, per GB costs for data continue to go down as volumes go up. At 20 TB/day with a year of hot data retention, the total cost of ownership (TCO) per GB cost for a year of hot data retention in Hydrolix using AWS is just $.29/GB. This includes all costs (compute and storage) related to ingesting, analyzing, and storing the data for a full year. And this doesn’t include any extra volume discounts that you may negotiate with your cloud provider. You can also opt to shorten retention periods to reduce your storage and compute costs if you don’t need to keep all your data for a full year, lowering TCO further.
Comparing the Data Retention of Hydrolix vs Elastic Cloud
Rising log volumes, cost, and data retention are all closely related. As log volumes and costs go up, teams look for ways to bring costs back down. That often means moving data into cold storage where it’s less accessible or throwing it away altogether.
See our infographic about 4 observability anti-practices that are really just throwing your data away.
To manage costs, Elastic introduced tiered storage with version 7.10 in 2020, which includes hot, warm, cold, and frozen tiers. While data in cooler tiers costs less, it’s less accessible for querying. Hot and warm storage tiers in Elastic Cloud are both too expensive for long-term retention, while cold and frozen storage tiers typically require rehydration.
It’s common for data in cold and frozen tiers to turn into dark data because it’s inconvenient to query. In this situation, it’s typically only kept for compliance or for forensic analysis, such as when a major breach is discovered. While cold data is much more cost-effective, if you are storing petabytes of data that you can’t readily query, you’ll still be spending a lot of money for minimal value in return.
To address this issue, Elastic introduced searchable snapshots in version 7.11, which was released in 2021. Searchable snapshots allow users to query data in the frozen tier more efficiently. However, there is a massive catch—it is only available for enterprise accounts.
The infographic in the previous section presents a hot-frozen scenario that uses searchable snapshots, but it’s still too expensive—leaving teams with the same issue as before. Even with the addition of data tiers and searchable snapshots, it’s prohibitively expensive to keep large volumes of data readily available for queries. As a result, enterprises will still have the same issues with short hot data retention windows.
Ease of Operations
Many systems are challenging to scale, have issues with overprovisioning, and can’t easily be scaled down after major events.
In the case of Elasticsearch, you need to consider sharding, indexing, data lifecycle management, and much more. Too many shards can lead to issues like hotspotting and inefficient indexing. Too few shards can negatively impact performance and lead to lost data if shards fail.
So what’s the right number of shards? As Elastic’s documentation states: “Unfortunately, there is no one-size-fits-all sharding strategy. A strategy that works in one environment may not scale in another. A good sharding strategy must account for your infrastructure, use case, and performance expectations.” Meanwhile, resizing shards typically comes with reindexing, a resource-intensive process that can slow down other critical operations.
As you scale up, you’ll need to adjust sharding and indexing. And due to Elasticsearch’s replica strategy, as you create more shards, you’re also creating replicas of those shards, leading to even more maintenance. The challenges of scaling up make Elasticsearch a poor choice for spiky traffic such as major events and unexpected outages.
Because Elasticsearch uses tightly-coupled architecture, components need to scale up together, which often leads to overprovisioning. There can be issues with resource contention, and even Elasticsearch’s own processes (such as reindexing) will compete for resources.
What about scaling down? If you’ve scaled up for a major event that has more traffic than usual, ideally you should be able to scale down afterwards to save on costs. Unfortunately, this process is complicated with Elasticsearch. When you remove a data node in Elasticsearch, all of its data must be transferred to other data nodes. This is a resource-intensive process and can lead to cascading failures if not done correctly.
In contrast, Hydrolix uses stateless Kubernetes architecture and each component can autoscale independently or be scaled manually as needed. You can quickly scale up for peak traffic or down for increased cost savings.
Hydrolix also automatically partitions data by time and has a background compaction service that optimizes data over time without additional configuration needed. And because all data is hot, there are no data tiers to manage.
Next Steps
If you’re interested in learning more about Hydrolix, contact Hydrolix about a managed trial or demo.
If you’re already an Elastic Cloud user and want to see how Hydrolix compares, our team can help you set up a Hydrolix cluster during a managed trial alongside your Elasticsearch clusters. By scaling up your cluster during the trial process, you can derisk the migration process and switch over to Hydrolix without any interruption to your data—and even continue to use Kibana.
Hydrolix can be used in place of Elastic Cloud for high-volume log data use cases and can also be used to complement your existing Elasticsearch clusters, whether you’re using Elastic Cloud or the open source ELK stack.
