Looking to reduce your Splunk bill? Start with this in-depth comparison of Cribl vs Hydrolix to cut Splunk costs, or combine them. The choice is yours.
Many teams love using Splunk, but rising data volumes are leading to big observability and cybersecurity bills, along with pressure to reduce costs. Fortunately, there are multiple solutions that can help you dramatically reduce your Splunk bill. This post will look at two leading solutions, Hydrolix and Cribl, how they can reduce your Splunk bill, and when to consider each solution.
Related reading 👉
Six proven strategies to reduce Splunk costs by 50% or more while keeping visibility into your log data for security and operations.
For starters: the cost to ingest data in Cribl Stream is $.32/GB, even if that data is sampled and thrown away. Meanwhile, the per GB cost of Hydrolix starts at $.20/GB with both bring your own cloud (BYOC) and bring your own storage (BYOS) options in Hydrolix. This pricing includes streaming data ingest, three months of data retention, and queries. It’s more expensive to use Cribl Stream to throw your data away than it is to keep it in Hydrolix.
Overview: Cribl vs Hydrolix for Reducing Splunk Costs
Cribl can be a good choice if you’re sampling and filtering data to reduce Splunk costs. If you want to send data to many different sources, Cribl Stream is a powerful tool for routing data wherever it’s needed. Meanwhile, Cribl Lakehouse and Lake offer flexible storage, though they come with tradeoffs. Lakehouse is comparatively expensive while Cribl Lake will have slower queries. If you want to analyze Cribl data in Splunk, Splunk needs to re-ingest the data first, leading to additional ingest costs. And once you surpass Cribl Stream’s free tier, the cost for ingesting data using Cribl can add up quickly.
Hydrolix is the better choice if you want complete data and your primary goal is to reduce your Splunk bill with no tradeoffs in the fidelity of your data. It uses techniques like high-density compression and optimized object storage to reduce Splunk costs, not sampling. Like Cribl, it includes a streaming pipeline where data can be preprocessed, but it’s designed only for sending data to Hydrolix. All data remains “hot” for querying (lakehouse performance at the cost of data lake storage). And with Hydrolix Search for Splunk, you can query your Hydrolix data directly in Splunk. Hydrolix doesn’t include native sampling, but you can use the Cribl integration with Hydrolix to sample data as needed. Ultimately, you can achieve the same cost savings as you would with Cribl, but without sacrificing your data.
Pricing: Cribl vs Hydrolix for Full-Fidelity Splunk Data
Let’s take a look at an increasingly common scenario for many enterprises: 2TB/day of telemetry data. Teams ingesting this volume of data into Splunk can have huge bills. For instance, a 600GB/day license for Splunk can cost enterprises more than one million dollars per year. In comparison, both Hydrolix and Cribl can save you a lot of money, but when it comes to full-fidelity data and fast access, Hydrolix can save you considerably more.
Here’s the baseline assumption:
- 2 TB/day of full-fidelity data
- 30 days of retention in Cribl, 90 days of retention in Hydrolix
- Fast access (Cribl Lakehouse versus Hydrolix)
Cribl Costs at 2TB/Day
When ingesting data in Cribl, you need to pay a flat ingest rate per GB in addition to paying for the underlying infrastructure.
Cribl Stream – Enterprise
The flat rate for ingesting data using Cribl Stream is $.32/GB for the enterprise level. For 2TB/day, this comes to approximately $19,200/month. To keep the infrastructure estimate as simple as possible, let’s assume that the flow of telemetry data doesn’t vary. At 2TB/day, throughput would average out to 23MB/second. Cribl infrastructure pricing is offered in 12MB/second increments, and the cost of 24MB/second is 3.9 credits/hour, or $3.90 per hour. That’s $93.60/day, or $2,849/month.
Cribl Lakehouse
To ensure fast queries, you can store the data in a medium-sized Cribl Lakehouse. The medium size has a 1200GB/day capacity. This is based on the amount of data stored post-compression, not ingestion, so the medium size is appropriate. Since Cribl achieves fairly significant rates of compression, some of that capacity will be wasted, but you have resilience if data spikes unexpectedly. The cost for a medium Cribl Lakehouse is $17,327/month.
Altogether, the total monthly cost of Cribl at 2TB/day (and fast queries) is $39,376.
You do have multiple cost-control levers at your disposal (such as sampling and using a slower data lake instead), though these levers do result in compromises in terms of the fidelity of your data and query performance.
Cribl Lake
You could, for instance, route all the data to Cribl Lake instead of Lakehouse. You’ll no longer have real-time analytics capabilities, but costs will be much lower. At a rate of $.05/GB for compressed data, and assuming 5x compression rates, the rate per raw GB would be $.01, reducing the cost of storage to approximately $600.
The total monthly cost using Cribl Lake instead (with compromises in terms of query speed) is $22,649.
Self-Managed Cribl Stream
Finally, you do have the option to use fully self-managed Cribl Stream in your own infrastructure, but the price is much higher: $84,000 for 1TB/day for the enterprise tier. The 2TB/day price is unlisted, but is likely significantly higher.
Data Routing to Splunk
If you route your data to Splunk, you’ll still pay $.32/GB (plus infrastructure costs) for every GB processed in Cribl Stream. Then you’ll have to pay any ingest and license costs for Splunk. So you could use Cribl Stream to sample just ⅓ of the incoming 2TB/month before passing it along to Splunk. That 600GB/month in Splunk may still result in a yearly bill of one million dollars. Plus you’ll be paying more than $10K/month just for Cribl Stream to throw the rest of the data away.
You could use Cribl Edge to reduce the amount of data sent to Cribl Stream for processing, but it’s yet another tool, more infrastructure, and additional costs.
Hydrolix Costs at 2TB/Day
With Hydrolix, you can either bring your own cloud (BYOC), which means Hydrolix runs in your own infrastructure, have Hydrolix host the infrastructure while you store data in your own storage for full control (BYOS), or even choose fully-managed Hydrolix. This post will discuss BYOC and BYOS options. The pricing estimate starts at $.20/GB on AWS infrastructure in the US east region. Both of these options include three months of data retention. Because of the way Hydrolix is designed, all data is “hot” for queries on object storage, giving you lakehouse performance regardless of the age of your data.
The total monthly cost of Hydrolix at 2TB/day (on AWS US east infrastructure) starts at $12,160/month.
Just as with Cribl, you have many cost-control levers at your disposal, and these will be covered in the next section.
With these assumptions, the cost of Cribl for full-fidelity data with fast access is ~3.2x more expensive than managed Hydrolix.
Even if you compromise on real-time analytics and use Cribl Lake instead of Lakehouse, the cost is more than 50% greater with Cribl than Hydrolix for full-fidelity data. And as discussed previously, the per GB cost just to process data in Cribl Stream (even if it’s discarded later) is more than the cost to ingest, store, and query your data in Hydrolix.
Ultimately, Cribl and Hydrolix can both help you cut your Splunk bill, but when it comes to full-fidelity data, you can reduce your costs much more with Hydrolix.
Querying and Visualizing Data
What about querying and visualizing your data? If you send it elsewhere, will your teams still be able to use Splunk for analytics, or will they need to learn new tools, leading to more tool sprawl and potentially increasing friction? Let’s take a look at how you can query your Cribl and Hydrolix data in Splunk. Then we’ll compare the native analytical capabilities of Cribl and Hydrolix.
Querying and Visualizing Your Data in Splunk
For teams that prefer using Splunk, Splunk Search Processing Language (SPL) is a powerful tool for filtering data using pipes, and Splunk includes excellent visualization and dashboard capabilities. It’s possible to analyze both Cribl and Hydrolix data in Splunk, but with Cribl, the process is more involved and costly.
Cribl Lake and Lakehouse
In the case of Cribl, there is no native integration that allows you to query your Cribl Lake or Lakehouse data directly in Splunk. Instead, you have to first filter the data in Cribl and then send it to Splunk to be ingested. Fortunately, Cribl doesn’t charge for egress (sending data from Cribl to Splunk), but you will be paying to ingest your data twice: once to Cribl when you initially ingest it, and then again to Splunk when you send filtered data there. This really limits your ability to use Splunk to analyze most of your data, as doing so will incur high costs and defeat the purpose of using Cribl in the first place.
However, it can still be helpful for use cases like forensic analysis. For example, if you have an incident or a breach, you can filter data for that time range in Cribl and then send it to Splunk for further investigation. This is akin to rehydrating data, and it can potentially be a time-consuming process, especially if you have different teams managing Cribl and Splunk. For instance, an ops or cybersecurity team might have to put in a ticket with IT to send data over to Splunk, discouraging all but the most urgent investigations.
Hydrolix
With Hydrolix, you can query your Hydrolix data directly in Splunk using Hydrolix Search for Splunk, which is available for download from Splunkbase. Your teams don’t have to make compromises when it comes to using Splunk for analytics, and you don’t incur any ingest costs. As a result, Hydrolix is a more cost-effective, convenient approach if you want to continue using Splunk for analytics.
See examples of analyzing Hydrolix data in Splunk, including high-volume use cases such as SIEM and CDN logs.
The following image shows an example of using Splunk to query (using the hdxsearch command) and visualize SIEM data stored in Hydrolix.
Native Analytical Capabilities
Cribl and Hydrolix have completely different approaches to querying data. Cribl Search is designed to search data wherever it’s stored. Meanwhile, Hydrolix is designed to make data stored in Hydrolix available for search in the tools your teams prefer. Let’s take a look at both approaches.
Cribl Search
Cribl Search uses Kusto Query Language (KQL) to search across structured, semi-structured, and unstructured data. KQL is designed to be easy to use and has many similarities to SQL. Wherever your data is stored, you can use Cribl Search to query it in one place, a great example of how Cribl provides flexibility to teams. This approach to federating data can be especially convenient when you need to analyze data across many sources. However, teams really do need to be careful when querying across sources, especially when data is semi-structured or unstructured. While KQL is inherently versatile and performant, it won’t be nearly as effective if data isn’t well-structured. That’s where it’s important to use Cribl Stream to standardize and structure data as much as possible. If you’d rather not use Cribl Search, you can route data to other solutions (such as observability platforms) and you can also use Grafana. For routing data to other solutions, you won’t have egress costs, but you will likely incur ingest costs in the platform.
Hydrolix
In contrast to Cribl, Hydrolix uses an ANSI-compliant SQL interface that includes elements of ClickHouse syntax. It is easy for SQL users to learn. Hydrolix ships with Grafana, but is also compatible with other visualization tools such as Superset and Kibana, and that includes support for Kibana Query Language (the other KQL).
Unlike Cribl Search, Hydrolix is specifically for querying Hydrolix data, not data from other sources. Instead, Hydrolix has connectors so that you can query your Hydrolix data in other interfaces. That includes platforms like Splunk as well as platforms compatible with Apache Spark such as Databricks, AWS EMR, and Azure Fabric. With this approach, you can make your telemetry available for other use cases such as business intelligence and machine learning. This broadens the value of telemetry beyond observability and cybersecurity.
Cost Flexibility and Control
Both Cribl and Hydrolix give you tremendous control when it comes to reducing costs. They share some key similarities in terms of providing flexibility to transform and shape your data before it’s ingested into storage. But they also have some significant differences. Cribl’s strengths are the ability to route data where you need and sample as much as you want. Meanwhile, Hydrolix provides control over your infrastructure, allowing you to adjust compute resources and reduce cloud costs. Ultimately, the central premise of Cribl is to control the volume of data to reduce costs, while the central premise of Hydrolix is to dramatically reduce the cost of data at scale so you can keep it all if needed.
Streaming pipelines
Both Cribl (through Cribl Stream) and Hydrolix offer real-time streaming pipelines for high-volume data. Both are scalable, with Cribl having the flexibility to route data to different sources. More complex pipelines and constraints at destination sources can cause latency issues, but this is not a flaw in Cribl Stream itself.
In contrast, Hydrolix has a real-time streaming pipeline that supports ingesting and storing .hdx format data in commodity object storage with any major cloud provider. Time to glass, even for high-volume events like the Super Bowl that generate hundreds of terabytes over the course of a few hours, is approximately 5-10 seconds.
Transformation capabilities
Both Cribl and Hydrolix offer a full suite of transformation capabilities that you can use to control and shape incoming data. That includes standardizing fields, redacting data, creating and evaluating new columns on the fly, dropping columns, calculating aggregations, and more complex transformations.
Data sampling
One specialty of Cribl is sampling data. Sampling rules can be scoped based on specific fields. For example, you could easily filter out DEBUG-level and other logs that are generally considered to have lower value. In contrast, Hydrolix doesn’t include native sampling, though you can use its integration with Cribl to stream and sample data into Hydrolix.
Hydrolix indexes all columns by default, partitions data by time, and uses high-density compression to achieve compression rates that are typically 10x-50x. These optimizations allow for real-time analytics on object storage and result in extremely cost-effective, long-term retention.
Infrastructure and Storage
Hydrolix
When it comes to controlling costs, Hydrolix offers significant levers for price control in the area of infrastructure if you take a BYO C+S. All systems (ingest, storage, and query) are decoupled and autoscale as needed. You can also create separate pools (such as separate query pools) with different levels of compute allocated. For example, you can scale up compute resources for an ops team query pool during an incident, and you could keep compute resources low in a query pool for training ML models. You can also scale any infrastructure to zero to save on costs. Other features such as circuit breakers allow you to constrain compute on queries as needed. Alternatively, by letting Hydrolix manage compute for you, you’ll still achieve tremendous cost savings without any additional overhead.
With Hydrolix, you pay for your own object storage when you bring your own cloud, which may be the list price (currently just over $.02/GB), or, if you’ve negotiated cloud-provider discounts, could be lower. And with high-density compression reducing the size of raw data by 10x-50x, each terabyte of storage is effectively just a few dollars per month. Due to these extremely low costs, Hydrolix offers 15 months of retention by default, though you can choose how long you retain your data.
Cribl
For Cribl, your levers will mostly be in the areas where you route your data. For Cribl Stream, you can scale worker pools up or down to save on infrastructure costs, but you’ll always pay the flat ingest fee per GB. In terms of storage and analytics, you can choose to route data to Cribl Lake or other cost-effective storage instead of the more costly Cribl Lakehouse.
Cribl Lake also has the option for direct ingest of Splunk data. This ships data directly from Splunk to Cribl Lake without any processing from Cribl Stream. It is slightly more expensive than Cribl Lake ($.10/GB instead of $.05/GB) but you don’t have to pay $.32/GB for ingest. In the short term, this can provide a lot of savings, but it’s only for extending the retention period of Splunk data without keeping it in Splunk. For long-term retention the higher cost of direct ingest will eventually end up costing more than Cribl Lake.
Hydrolix and Cribl: Two Approaches for Reducing Your Splunk Bill
Cribl and Hydrolix take completely different approaches to reducing your Splunk bill. Cribl allows you to preprocess and intelligently sample and filter your data before you send it to Splunk or other sources. If you want to filter the majority of your data, then Cribl is a great approach. It also offers an intuitive UI and a full suite of products, including storage and analytics, though Cribl Stream remains its flagship product.
Meanwhile, Hydrolix drastically reduces the cost of ingesting, analyzing, and storing telemetry data, making it possible to keep full-fidelity data long-term. It doesn’t include native sampling, but it’s so cost-effective that you’d need to have a high rate of sampling in Cribl just to achieve the same level of savings. With Cribl Stream and Cribl Lakehouse costing approximately 3x as much, you’d need to have a 30% sampling rate just to make the cost equivalent. In other words, you need to throw away 70% of your data just to achieve the same level of cost savings.
Ultimately, you don’t need to choose between Hydrolix and Cribl. Both are excellent solutions depending on your use case. You can use Cribl Stream to route data wherever you need, including to Hydrolix. The choice is yours. You get flexibility, full control of your data, and tremendous savings.
Next Steps
- Explore our pricing calculator to see how much you’ll save.
- Read about our integration with Cribl.
- Try a proof of concept or demo.
How can I reduce Splunk costs without sampling or discarding data?
Try offloading high-volume, lower-value telemetry data to a data lake that utilizes commodity object storage. By moving data to a more cost-effective storage layer while keeping it queryable, you can avoid the visibility gaps caused by traditional data sampling or filtering methods.
Is it possible to query offloaded data using standard Splunk SPL?
Yes. You can analyze external data using familiar workflows and languages through dedicated integration apps available on Splunkbase. With these integrations you can use SPL commands to query stored data directly from Splunk.
How does high-density compression impact observability infrastructure bills?
It shrinks the physical size of raw log data, dramatically reducing storage. Organizations pay for significantly less storage capacity and reduced data transfer overhead. Cloud infrastructure costs decrease as a result. And, because the data is smaller, it requires fewer compute resources to scan and retrieve during queries. This provides a dual layer of cost savings across both storage and processing vectors.
