It’s the time of year for predictions, when bloggers everywhere look into their crystal balls (or maybe ChatGPT) to come up with top ten lists of what to watch for in the new year. Their predictions will sometimes be wrong or obvious, with some true nuggets of wisdom standing out.
None of these predictions, no matter how good or on point for your organization, will be as useful as the projections you can make with your own data.
And here’s a prediction that’s very likely to come true. The volume of data your organization handles will continue to go up this year—quite possibly by a lot. On average, enterprises saw a 5x growth in their log data over the three year period ending in 2023. More than 20% of those enterprises saw at least 10x growth. There’s no reason to believe this trend is slowing.
Because of the huge growth in data, many organizations are dealing with prohibitively high costs, leading to tough decisions. To flatten the cost curve, they’re often forced to choose between their data and their bottom line. And you don’t need to read tea leaves to predict that the bottom line usually wins. The result is gaps in data that can make it much harder to find and quickly fix issues, detect security breaches, and train machine learning models.
These gaps also make it much more challenging to make data-driven decisions about the future of your business. Or to give your data science and business intelligence teams the material they need for forecasting mission-critical issues like inventory, capacity, and product demand.
How do you avoid these gaps? Future-proof your data by choosing a solution that allows you to retain data cost-effectively, without compromising query performance or having to handle frozen data. Let’s look at what this means in practice.
Discarding Data Isn’t the Answer
If you’re still using a solution (whether for observability, security, or another use case) that uses tightly-coupled storage and compute to provide real-time performance, then your organization is already at risk—and you may already be dealing with high costs and compromised data quality.
Discarding data is at best a short-term solution for saving money. Unfortunately, many organizations are forced to discard their data or engage in anti-practices that reduce the granularity of their data. These anti-practices include:
- Downsampling: Only a percentage of the data is retained while the rest is thrown away. This sample could be random or it might involve only collecting log lines that meet certain thresholds. This approach is especially common with many observability platforms, and high-volume data such as traces are often sampled. But with downsampling, you’ll end up with huge gaps in your data. These gaps might include information about mission-critical issues such as security breaches.
- Aggregating (but not keeping the raw data): In this scenario, data is typically rolled up into summaries. For example, you might retain averages and sums for daily, weekly, and monthly reports. While summarized data can be extremely valuable, organizations often throw away the underlying raw data. As a result, teams are no longer able to drill down into the data or aggregate it in different ways to answer new business questions.
- Moving data into tiered, difficult-to-access frozen storage where it goes dark: Frozen storage can help flatten the cost curve, but this data usually becomes dark data. Data goes dark because it’s difficult to access. This is typically an issue with frozen storage, where data needs to be rehydrated for analysis, a time-consuming process.
- Throwing data away altogether: This may seem like the worst option of all, but it’s surprisingly common.
None of these solutions will future-proof your data. They aren’t even going to be very effective at keeping costs down long-term. For example, in the case of sampling, as the volume of data goes up, you’ll need to downsample that data even further to flatten the cost curve.
If you’re sampling 10% of your logs today and the volume of your logs doubles in the next year (a common scenario for many businesses), you’d need to reduce the sampling rate to 5% to keep costs down. Eventually, you’ll be sampling such a small percentage of the data that its value will be minimal. Many organizations have already reached that point.
Meanwhile, enterprises spend an average of 52% of their storage costs on dark data. While the majority of that data is kept in more cost-effective frozen storage, it often has negligible value. It’s too inconvenient to rehydrate, and as a result, it isn’t useful for data-driven decisions that can help your business.
Transforming the Economics of Data Storage
To be effective for both real-time and historical data analytics, log storage solutions must:
- Have the capacity to ingest terabytes of data everyday
- Store petabytes of data while remaining cost-effective
- Efficiently query all that data
Traditionally, it hasn’t been possible to combine all three of these features in a single solution. You could effectively choose two of these three options: scale, performance, and cost-effectiveness. Solutions that were both performant and cost-effective just five years ago didn’t encounter the scale that is increasingly common today. But as we’ve seen, reducing the scale (typically by discarding or sampling data) of data in order to retain performance and some degree of cost-effectiveness doesn’t work.
Enterprises using the cloud are increasingly turning to decoupled object storage to solve the problems of cost and scale. But while solutions like AWS S3 provide cost-effective, scalable storage, they aren’t designed for high-performance analytics. The recent release of S3 Tables is a step in the right direction, and a new generation of platforms like Hydrolix are maximizing the performance of decoupled storage in order to give enterprises the ability to ingest, store, and query petabytes of data—all while remaining performant and cost-effective.
If you’re currently using a solution that doesn’t include decoupled storage—or if you’re already dealing with high costs and making compromises when it comes to the quality of your data—then 2025 should be the year you migrate to a new solution that solves this fundamental problem so you can focus instead on solving the problems that are specific to your business.
And if you’re just now beginning to evaluate platforms, it’s important to think not just about the amount of data your enterprise is ingesting and storing today. You also need to consider how much you’ll need to store and retain. And that’s not just to next year but to 2030 and beyond. Remember, a 5x-10x increase in the amount of data isn’t an outlier.
Now is the time. Make it an organizational goal in 2025 to find a solution that saves you from falling into log data gaps, without breaking your budget.
