When it comes to working with large volumes of log data, Elastic Cloud costs too much. There’s no good way to make the math work, even with data in the frozen tier—or even if you discard data altogether. If you’re ingesting at least one terabyte of log data per day, Hydrolix can maximize the value of your data with long-term hot data retention while dramatically reducing your costs.
Let’s look at some pricing scenarios using Elastic’s pricing calculator, and then compare the costs against Hydrolix’s pricing estimator. All of these estimates assume an ingest volume of one terabyte per day and retention of one year. The savings are even better at higher volumes of data.
We’ll look at three Elastic Cloud scenarios that include seven days of hot data retention, with data either moved directly to frozen storage or thrown away. Next, we’ll compare that to Hydrolix’s costs. Then we’ll take a deeper look at each of the scenarios and the underlying assumptions. Or you can just check out the infographic to see the difference!
Our scenarios:
- FOMO (Fear Of Missing Out): 1 week hot data retention with Elastic Cloud plus a year of frozen storage with searchable snapshots.
- CYA (Cover Your A**): 1 week hot data retention with Elastic Cloud plus a year of frozen storage with no searchable snapshots (data must be rehydrated from frozen storage for queries).
- YOLO (You Only Live Once): 1 week hot data retention with Elastic Cloud and then throw the rest away!
- OMG (Oh My God): 1 year of hot data retention with Hydrolix.
TL;DR: No matter how you do the math, Elastic Cloud is much more expensive than Hydrolix. At 1 TB/day with data accessible for querying using searchable snapshots, Elastic Cloud is at least 7x more expensive than Hydrolix. And even if you discard data after just seven days in Elastic Cloud—compared to keeping it “hot” for queries in Hydrolix for a year—it’s still 3x more expensive.
FOMO (Elastic Cloud With Searchable Snapshots)
In this scenario, you have only seven days of hot retention in Elastic Cloud, but you want to query the rest of your data whenever you need to so you don’t miss out on valuable insights.
That means seven days hot and a year of frozen storage with searchable snapshots, which allow you to query snapshots in frozen storage without first mounting all the data. Searchable snapshots are only available with an enterprise account ($$$). And while searchable snapshots are more efficient than rehydrating your data, they aren’t going to give you hot tier performance.
Cost: At $102.4308/hour, the cost is $74774.48/month.
CYA (Elastic Cloud With Frozen Storage)
If FOMO is too expensive, you might be willing to settle for CYA. This includes seven days of hot storage and a year of frozen storage with no searchable snapshots.
Without searchable snapshots, you can get by with a standard account (provided that there aren’t other gold/platinum/enterprise extras that you need). A standard account costs a lot less than an enterprise account, but…
Almost all of your data will be frozen and must first be rehydrated for querying. Data won’t be queried unless you really need it, which means you’ll have a lot of dark data. At least you’ll be able to rehydrate it if something bad happens.
Cost: At $52.2720/hour, the cost is $38,158.56/month.
YOLO (Elastic Cloud With Data Discarded After Seven Days)
You only live once, so it’s all about living in the moment. Seven days of hot retention in Elastic Cloud and then it all gets thrown away. Because if you didn’t see it, it didn’t happen.
You’ll save over CYA, but not by as much as you might hope considering that you have to throw away all your data after a short period of time, which can lead to all kinds of issues with compliance, threat hunting, business intelligence, and other use cases.
You might be surprised how many enterprises are forced into YOLO situations by the high costs of Elastic Cloud.
Cost: At $38.4480/hour, the cost is $28,067.04/month.
OMG (Hydrolix With 12 Months of Hot Data)
Finally, we get to the OMG—what you get with Hydrolix in comparison. Keep all your data as long as you need it. And all data is hot for low-latency queries regardless of whether it’s a minute or a year old. There are no tiers that involve additional operational overhead, only to give you substandard query performance.
Hydrolix comes with 15 months of data retention by default, but we’ll set it to 12 to compare it with the Elastic scenarios.
Cost: $10,423/month.
The Final Verdict: Dramatically Reduce Costs With Hydrolix
Elasticsearch is a powerful tool for many use cases, and for some use cases, Elastic Cloud can be a good managed option for Elasticsearch.
Unfortunately, Elastic Cloud is simply too expensive for large volumes of log data. And with log data volumes continuing to go up, Elastic Cloud isn’t the right choice for terabyte-scale volumes in use cases like observability.
Instead, for log data use cases of at least one terabyte per day, Hydrolix is much more cost-effective. For the scenarios described above, Hydrolix is one third of the cost of the YOLO option—and comes with a year of hot data instead of just seven days.
Many enterprises are already ingesting more than one TB/day (some Hydrolix users ingest more than 30 TB/day). At higher scales, the cost of Elastic Cloud is even more prohibitive, forcing enterprises to adopt yet shorter hot data retention periods. And as the YOLO plan shows, throwing away data may cut your costs, but you’ll still be paying too much for Elastic Cloud. The math just doesn’t work out.
If you’re working with high volumes of log data, Hydrolix is a better choice than Elastic Cloud, at least when it comes to costs. You can even use Kibana with Hydrolix, allowing your teams to keep the visualization and query tools they know and love.
Learn more about Hydrolix’s Quesma integration, which allows you to use Kibana with Hydrolix data.
A Deeper Dive Into the Pricing Scenarios
Let’s take a look at how we came up with the numbers for these scenarios. That means taking a quick look at Hydrolix’s pricing estimator, then taking a deeper dive into Elastic Cloud’s pricing calculator, which is much more complex.
Hydrolix offers a simple pricing estimator with just three inputs: raw data per day (TB), cloud vendor, and data retention. For this post, the assumptions are 1 TB/day, AWS (Elastic Kubernetes Service), and a data retention policy of 12 months.
In contrast, Elastic Cloud’s pricing calculator has a lot of inputs. So here’s how you can replicate the numbers in Elastic’s pricing calculator (as of November 2024).
We chose AWS as the cloud provider because AWS has the most market share of the three major cloud providers (AWS, GCP, Azure). Pricing can be considerably different with other cloud providers. For region, we stick with the default: Ohio (us-east-2).
Next, let’s consider the hardware profile required for real-time analytics. Elastic Cloud’s pricing calculator describes a hardware profile as a “unique blend of virtual storage, RAM, and vCPUs.”
Typically, real-time analytics requires running a fair number of dashboards and queries with significant spikes during major events. So we chose general purpose (ARM) storage, which Elastic Cloud describes as suitable for use cases requiring 5-7 days of query retention (exactly what we have in these scenarios) as well as search use cases with moderate loads and occasional high query loads.
You can save on costs by using storage optimized hardware profiles, but those are recommended for “light” search use cases. If you’re running lots of dashboards and queries, especially during a major event, you’ll want a profile that has better query performance.
For the hot tier, each of the Elastic scenarios uses 7.03 TB storage | 480 GB RAM | 128.1 vCPU. That perfectly handles 7 days of hot retention at 1 TB/day. Of course, that .03 TB doesn’t leave much room for overages and assumes that ingest remains consistent. And with log volumes going up, the storage required will probably need to be resized on a regular basis. Contrast this to Hydrolix, which uses S3-compatible storage that’s easy to scale.
Elastic recommends 3 availability zones for mission-critical use cases. You might be able to get away with just 2 availability zones (the minimum recommended for production use cases), but this would go against Elastic’s own recommendations.
For the frozen tier, these assumptions use 375 TB storage | 240 GB RAM | 31.9 vCPU. Assuming 1 TB/day with 7 of those TBs in the hot tier, you’ll have 358 TB in the frozen tier from a year’s worth of data. 375 TB is the lowest Elastic Cloud storage capacity that accommodates 358 TB.
Since the frozen tier will likely be queried less (especially for the CYA tier), these assumptions use 2 availability zones. Could you potentially get away with one considering that S3-compatible storage has its own availability and resiliency guarantees? Maybe, but that would be going against Elastic’s recommendation of at least 2 availability zones for production.
At this level of scale, Elastic Cloud requires master instances. These assumptions use the default size per zone: 4 GB RAM | Up to 8.5 vCPU and 3 availability zones.
Finally, these assumptions include Kibana instances sized at 8 GB RAM | Up to 8.5 vCPU across 3 availability zones. After all, you don’t want your Kibana instances to go down when you most need them, so high availability is required.
These assumptions don’t include Integrations Server instances, which are required for observability and security data from Elastic agents as well as APM. So if you want this data (and you probably do), that’s an extra cost.
And as your log data volumes continue to grow, the price of an Elastic Cloud deployment will get even more prohibitive, while the per GB cost for your data will go down in Hydrolix.
Next Steps
- See the pricing differences yourself by comparing Hydrolix’s pricing estimator to Elastic Cloud’s pricing calculator.
- Learn how you can use Kibana with Hydrolix, making migrations from Elastic Cloud to Hydrolix easier for your teams.
- Ready to consider Hydrolix? Contact Hydrolix about a proof of concept or demo.
