Many enterprises are dealing with high costs when it comes to log analytics, leading to difficult choices. Do you pay high costs and break the budget or engage in cost-cutting measures that may compromise the success of your business? The answer should be neither. Log data is business-critical for a wide range of use cases, including observability, data science, and building better AI models. You shouldn’t be forced to choose between data and your bottom line.
Hydrolix consistently delivers the most cost-effective real-time log analytics for businesses dealing with the challenges of big data. This post will provide a high-level view of how Hydrolix can reduce the total cost of ownership (TCO) for use cases like observability by 4x, 6x, or even more, all while providing long-term data retention and fast queries for all your data regardless of age.
Broadly, the post covers two kinds of efficiency:
- Reducing storage costs through high-density compression and optimized object storage.
- Reducing compute costs through methods like autoscaling, high-efficiency queries, and summary tables.
The following video provides a quick overview of how Hydrolix dramatically reduces the cost of cloud object storage.
Reducing Storage Costs
The ability to store years worth of data, at full fidelity, all while keeping it “hot” for fast queries, is a cornerstone of Hydrolix. There are multiple design elements that make this possible. One of the key innovations behind this is high-density compression that can reduce data sizes by 20x or more, dramatically reducing storage costs.
The following dashboard shows the compression of hydrologs, a petabyte-scale dataset used internally at Hydrolix.
The grey rectangles show the raw volume of logs generated by the Hydrolix platform in the first six months of 2025. The tiny green rectangles on the top of the grey ones show the compressed volume of these logs. As the detail box shows, the data is extremely compressed. This date, chosen at random, shows that 398 TiB of data have been compressed to just 4.91 TiB, an 81x rate of compression.
Hydrolix compression allows enterprises to keep all their data more cost-effectively, at full granularity, with sub-second queries for all historical data whether it’s a minute or a year old. This opens up a world of possibilities that were difficult, slow, and expensive before. Here are just a few examples:
- Low and slow attacks can be detected with a higher level of confidence. Unlike volumetric attacks, these kinds of attacks show up occasionally over a long period of time. Detecting them requires querying detailed historical data.
- Time to market for AI/ML use cases is accelerated. AI models need data for development and refinement. More data leads to more accurate models. Meanwhile, faster queries reduce the time it takes to refine and improve models.
To demonstrate how much Hydrolix can reduce storage costs, the following diagram shows the reduction in cost of Hydrolix data in standard cloud object storage—in this case, AWS S3—as opposed to storing the data without compression in S3. Once again, the data is from hydrologs. The blue rectangles represent the cost of storage with Hydrolix compression—while the full-height rectangle (blue + white) represets the cost in AWS storage without compression.
Hydrolix doesn’t just store data in cost-effective object storage—its high rate of compression dramatically reduces storage costs, making it possible to retain petabytes of data long-term much more cost-effectively than even traditional data lakes. And unlike traditional data lakes, which aren’t designed for high-efficiency queries, Hydrolix data is structured for fast analytics.
Reducing Compute Costs
Compute and storage can lead to extremely high costs at scale, so making compute more efficient is another way to reduce total costs. Hydrolix is designed for extreme efficiency when it comes to compute. Here are just a few of the approaches that Hydrolix takes to reduce compute, and by extension, costs.
- Autoscaling: Infrastructure scales automatically based on demand, and there are no wasted resources due to overprovisioning. All systems are stateless and decoupled so they can be scaled independently with no resource contention between systems.
- High-efficiency queries: Techniques like microindexing and partition pruning reduce the amount of data that needs to be scanned and retrieved.
- Summary tables: Summary tables provide fast analytics on business-critical metrics and aggregates without needing to query the underlying raw data.
Let’s take a closer look at each of these.
Autoscaling
Many platforms offer fixed compute, which is inflexible and either leads to overprovisioning (and wasted budget) or poor performance. In contrast, platforms that autoscale can provide just the right amount of compute when it’s needed, maximizing performance without wasted costs. The following diagram illustrates this principle.
The blue line shows autoscaling to compute demand and accounts for spikes, lulls, and even reducing compute to zero during off-peak times where enterprises may want to save on costs. When you consider that Hydrolix also has decoupled systems and compute autoscales separately for ingest, storage, and query, there’s also no risk of overprovisioning one subsystem in order to provide enough resources to another subsystem.
The yellow line shows fixed compute that’s overprovisioned. To ensure that there is always enough compute for spikes, enterprises need to provision for the highest spike (and generally add additional compute just to be safe). In this situation, the required compute is always available, but there is often idle compute that leads to wasted resources.
In contrast, the gray line shows compute that’s underprovisioned. During spikes in demand, underprovisioning will lead to poor performance. And it will likely lead to some wasted resources as well because lulls may still need less compute than the underprovisioned capacity.
It can be extremely challenging to predict demand for provisioned resources. After all, use cases like observability are all about capturing and mitigating the unpredictable. A DDoS attack can lead to a massive spike in traffic, and if provisioning doesn’t account for the increase in traffic, operations teams will be dealing with poor performance when they most need visibility.
Many platforms and SaaS offerings use shared architectures and compute to prevent overprovisioning, but these platforms can still have issues with resource contention and outages. And the largest enterprises will often need dedicated instances and compute to account for considerations like data governance, security, and privacy.
High-Efficiency Queries
Hydrolix uses techniques like micro-indexing, partition pruning, and column pruning to minimize the amount of data that needs to be scanned and retrieved.
- All columns are indexed by default in Hydrolix using micro-indexing, a technique that allows users to request narrow byte ranges of data from object storage.
- Hydrolix partitions data by time, and time-filtered queries use partition pruning to scan only partitions that have the included date range.
- Hydrolix is a columnar datastore, allowing users to retrieve only the columns they need.
We’ve written extensively about how Hydrolix query infrastructure works. See these posts for more information on Hydrolix queries are designed for scale and efficiency:
- Maximizing Query Performance for 100+ Billion Row Data Sets
- Dissecting a Hydrolix Query
- Reduce Your Cloud Spend with Precision Query Scaling describes how users can use dedicated, separate pools for querying. This kind of precise separation of compute is also possible with other Hydrolix subsystems such as ingest.
Summary Tables
Metrics and aggregations are a business-critical part of many use cases ranging from observability to real-time analytics. However, querying and aggregating data directly from trillion-row raw datasets is extremely inefficient, both in terms of compute and performance.
Hydrolix uses summary tables to store and update metrics and aggregations in real time. These tables are typically at least 95%-98% smaller than the underlying raw tables. For example, instead of querying a table with ten terabytes of data and then aggregating the results, users instead might query a summary table that’s only 200 gigabytes (98% smaller), and the results are already pre-aggregated.
The next diagram shows how data is ingested into and queried from both raw and summary tables.
Data is inserted into both summary tables and the underlying raw data table at ingest time, so summary tables never need to query raw data or contend for resources with other users and analytics systems. Dashboards and queries can access summary tables for broader metrics and aggregations. Meanwhile, when stakeholders need access to more granular data, they can query the underlying raw data as needed, which remains “hot” for fast queries regardless of age.
Next Steps
- Check out our pricing calculator to estimate the total cost of ownership for Hydrolix and learn how much you can save over your current solution.
- If you’re interested in learning more about how Hydrolix can reduce the total cost of ownership for your specific use case, request a demo or start a free managed trial.
