Why All Your Data Storage Should Be Hot

If you’re working with large volumes of log data, the thought of runaway costs might make you break into a cold sweat. Your hot data costs too much to store while your cold storage is too difficult to access. You could try to fix the cost issue with warm storage, which costs less than hot storage but comes with increased query latency. You’ll need to manage tradeoffs such as decreased performance—and your warm storage still might cost too much. Or maybe you’re working with so much data that the only cost-effective solution is to put all your data on ice.

But what if you didn’t have to make that tradeoff between cost and performance? What if you could get hot storage access at the cost of cold storage? With Hydrolix, you get subsecond query performance from inexpensive cloud storage, eliminating the tradeoff between hot and cold data and making all of your data both easy to access and cost-effective.

This post covers:

Understanding Hot Data in Modern Infrastructure

Hot data refers to frequently-accessed data that needs to be readily available for querying. For some use cases, it refers to millisecond query latency, while for others (such as real-time analytics), query latency can be sub-second or in the seconds.

In order to keep data hot with millisecond latency, platforms must use closely-coupled (and expensive) storage solutions such as solid-state drives (SSDs), storage arrays, and in-memory storage. All of these solutions can potentially provide very fast query access, but the trade off is high costs—which quickly become prohibitive for the large volumes of data that many enterprises are dealing with today. This level of performance is necessary for certain use cases such as programmatic advertising (where ad placements must happen with millisecond latency).

However, for many use cases such as real-time analytics, observability, and business intelligence, daily data volumes can reach terabytes while long-term storage can hold petabytes of data. Modern data solutions are now maximizing the performance of object storage to provide cost-effective, sub-second query latency on petabytes of data. Hot data in this context is much more affordable—and it’s the only viable approach for many enterprises that need to keep costs down while dealing with increasing volumes of log data.

Hot Data vs Cold Data

In contrast to “hot” data, “cold” data is less accessible for querying because it’s stored in a more cost-effective storage medium that’s typically much slower for querying. It may be suitable for some use cases where data isn’t frequently accessed, but it comes with a significant trade off—it may be difficult to access, often requiring time-consuming rehydration before it can be queried.

Traditionally, “hot” data was simply data that was readily available for querying, while “cold” data was typically offline and stored on tape—it would need to be manually retrieved before it could be queried.

However, with modern data architectures, some platforms offer three or even more tiers for storage, with hot, warm, cold, and frozen being the most common terminology.

Hot data storage refers to data storage that requires sub-second or sometimes even millisecond query latency. Millisecond latency is extremely expensive for large volumes of data because it requires closely-coupled storage solutions that are more expensive and can only be scaled vertically. Meanwhile, solutions using object storage can provide sub-second query latency much more cost-effectively.

Warm data storage is a compromise between hot and cold data. It provides slower access than hot storage, but it’s faster than cold storage. It’s more cost-effective than hot storage, but more expensive than cold storage.

Cold data storage is considerably slower but also more cost-effective, while frozen storage data is even more affordable—but also more time-consuming to rehydrate for querying. Frozen data often contains archived data, and because it is difficult to access, it typically goes “dark”—no longer visible to the organizations storing it. When it’s rehydrated, it’s often only when something has gone wrong, and forensic analysis (such as with a data breach) is required.

Tiered data solutions can become increasingly convoluted, especially for teams that have to manage pipelines moving data from one tier to another. They are often presented as an effective solution for managing data—and handling the trade offs of data cost versus query performance. However, in reality, they can also be seen as band aids over a pervasive problem: many platforms are built on legacy storage systems that are closely-coupled and expensive at scale. These platforms may have incurred too much technical debt to replace their underlying architecture, and so they pass the costs and challenges of managing multiple tiers of data to users.

The following customer story shows one example where an enterprise dealt with increasingly large volumes of incoming data, leading to prohibitively high costs. The compromise was to discard data after a certain period of time. With a previous platform, the amount of time grew shorter and shorter—from months to a matter of days. When the enterprise switched over to Hydrolix, they were able to retain data for a full year. The longer retention allowed them to help their own customers maintain data for compliance standards.

Why Hot Storage Usually Costs So Much

Hot storage gives you efficient query performance, low latency ingest, and high scalability. However, these benefits have traditionally come with a cost. To achieve high performance, most platforms use expensive storage solutions like solid-state drives (SSDs) and storage arrays. These storage solutions offer fast read and write times for your data, but the extra cost makes your hot storage expensive—and this architecture makes it prohibitive for all of your storage to be hot.

These costs are even higher when you consider the sheer amount of data that many companies are ingesting on a day-by-day basis. To reduce costs, it’s typical to move real-time data to warm, cool, or cold storage after a certain amount of time has elapsed. These tiers come with tradeoffs such as lower query performance. For many log management solutions and observability platforms, a typical default for hot data storage might be somewhere between three and thirty days. Some companies ingest so much log data that it’s necessary to move data to cold storage after just a few days to be more cost effective. And some solutions won’t retain your log data beyond the hot storage time period, which means you have to export it to cold storage yourself, creating additional headaches and complexity.

The problem with this model is that hot storage retention periods simply aren’t long enough for complete data analysis. Data-intensive applications like e-commerce personalization, ad targeting, XDR, ML model training, and observability are just a few use cases where limits on data retention can negatively impact your business.

The Problems With Cold Storage

When your data goes into cold storage, it becomes much harder to compare, understand, and analyze. As the saying goes, “Those who cannot remember the past are condemned to repeat it.” The same could easily be said of your data. What happens when your team has a major incident that looks similar to something that happened six months or a year ago—but you can’t query data from that time period to compare and see what went wrong? Or what if you are trying to understand patterns in your data that suggest periodic slowdowns, but you can’t go back far enough to better understand those patterns?

Another typical use case where cold storage is problematic is for large scale, cyclical events. For example, you might have a yearly Black Friday sale or manage other large annual events such as holiday specials, sporting events, and more. When all your storage is hot, you can make comparisons across large events in near real time that can support your business. When the data from the last major event is in cold storage, though, you can’t make those connections fast enough to act on them. You lose valuable contextual information—such as what a customer bought or looked at last year—to connect to your most recent data. And the customer that almost made a purchase last year might be a near-miss this year, too. Even worse, you’ll never connect those dots and understand what happened.

Ultimately, having more data over a longer time period increases its accuracy and precision, makes it easier to detect patterns, provides valuable context, and gives you richer insights. By moving data into cold storage where it’s hard to query and access, you risk losing the information your organization needs to thrive.

Downsides of Tiered Storage

Here are some of the downsides of tiered storage:

Slow queries: Cold storage provides less RAM than hot storage while also typically having many more rows of data than hot storage. The low ratio of RAM to data leads to much slower queries, which makes it difficult to get actionable insights in a timely fashion.
Query bottlenecks: Your queries are only as fast as the coldest tier of storage you are querying. If you make a query that pulls from both hot and cold storage, you’ll be waiting on the slowest part of your system. Even if you are using a solution that asynchronously returns query results (giving you the data in hot storage faster), you won’t have a complete picture until you get the full response.
Resource-intensive migrations. In addition to taking considerable compute resources to move data from one tier to another, you also need to keep your data backed up while it’s being written to the next tier, so it takes up extra space during the migration process. That also means you need additional storage overhead to handle duplicate data during migrations.
Extra maintenance. With tiered storage, you have to manage multiple storage solutions, not just one. Depending on your implementation, each solution might work differently.

The main benefit of tiered storage is to cut down on costs—but if you can keep all your storage hot at no additional cost, there is minimal benefit to tiered storage for most use cases.

Why All Your Storage Should Be Hot

The traditional cost differences between hot and cold storage have created a false dichotomy, one where your most recent data is valued while older data—sometimes data that is only days or weeks old—is devalued. But that’s rarely an effective way to value your data. You don’t know which data points you’ll need until you make a query. And the efficiency of that query will be limited by the coldest tier of data that you are pulling data from. If some of that data is hot, some warm, and some cold, you won’t be getting any of the benefits of hot storage in the first place. You’ll just be footing the extra cost.

When all of your storage is hot, you get the following benefits:

Historical Comparison

Make historical comparisons for deeper data insights. Compare today’s data against last week, last month, or last year—with no penalty for querying cold storage and no lost data.

Real-Time Analysis

Real-time analytics for all your data. When all your storage is hot, your queries will always be high performance and low latency. You can add historical context to your real-time data, whether that’s historical data about a customer that’s on your site now or real-time security insights on an IP address associated with a device that last visited the site six months ago.

Reduced Storage Complexity

Eliminate the complexity of storage tiers. Having multiple storage tiers injects complexity into your system. You’ll either need to manage the migration of data between tiers or rely on a platform that handles that migration for you. You’ll need to back up your data before the migration to ensure data integrity in case there are issues with migrating your data, and you’ll need extra storage overhead to account for duplicate data during migrations. When all of your storage is hot, you don’t need to worry about migrating data between tiers.

Simplified Data Management

Keeping all storage hot can eliminate difficult decisions about data management. When you’re working with tiered storage, you have to make challenging decisions. When should your data be moved from hot to warm and finally to cold? Cost isn’t the only consideration—you also need to understand which data is essential for operations. These decisions become even more complicated as your applications scale. As your data volume and costs go up, do you try to save money by shortening the retention period for hot storage? What if your engineering teams need the data while your CFO says costs need to be reduced? These decisions can be anxiety-inducing and cause friction between teams. When all your storage is hot (and cost effective like cold storage), you have one less problem to worry about—and more time to spend elsewhere.

Enhanced Security and Compliance

Many enterprises keep security and compliance data in low-access cold or frozen storage. The idea behind this approach is that the data will rarely (if ever) be queried. However, there are multiple issues with this approach.

In the case of security, many attackers use a “low and slow” approach. In other words, they may attempt to breach systems over an extended period of time. A few log lines on their own may not appear suspicious, but taken altogether, may suggest a malicious actor. As one example, a security or operations team practitioner might see a log line that suggests suspicious behavior. It may have an IP address from an embargoed country, or it might have a payload in the request. A search of the IP address or payload likely won’t uncover results in frozen storage, making it harder to correlate suspicious activity.

In the case of compliance, some enterprises need to turn over records related to compliance on short notice. A common example that extends across many industries relates to cybersecurity insurance. This kind of insurance can cover the potentially disastrous costs of a data breach, but the companies that provide it often require documentation of compliance, such as logs that demonstrate regular security best practices like penetration testing.

There are also many use cases that are specific to industries. For example, telecommunications companies often must be prepared to retrieve historical data on short notice.

If you can eliminate the extra cost traditionally associated with hot storage, why would you ever put your data in cold storage again?

Common Hot Data Use Cases

Let’s take a look at a few common areas where hot data is mission-critical. These scenarios focus on hot data use cases that require sub-second latency, which typically covers most real-time analytics use cases.

Observability and monitoring: Operations teams and SREs use observability and monitoring to get a comprehensive understanding of how their systems are performing. Observability allows teams to quickly detect and fix issues, leading to lower mean-time-to-detect (MTTD) and mean-time-to-repair (MTTR). Observability and monitoring use cases can lead to terabytes of log data being ingested every day, especially when considering services that emit a lot of logs such as CDNs. By keeping data hot long-term, teams (and machine learning models) can analyze the log for larger patterns in their data, which is important for use cases like anomaly detection.
Cybersecurity: SecOps teams working with security data need hot storage access to logs both when data is ingested (for real-time analytics) and long-term (historical analytics). In the short term, cybersecurity teams need to respond in real time to issues like DDoS (distributed denial of service) attacks. Meanwhile, breaches typically take months or even years to detect (272 days on average according to IBM). Threat hunters and other cybersecurity professionals need to be able to act on hypotheses quickly to track down threats—even if an intrusion attempt happened many months ago.
Machine learning: Machine learning models typically benefit from training on larger data sets—and long-range, high-fidelity data can be particularly useful for detecting both trends and outliers. However, models need readily accessible data for training. By keeping all data hot for querying, models have more training data, leading to higher accuracy and better predictive behavior.
Media and streaming: Major broadcasters and stream providers need to provide high-quality streams and positive experiences for end users—but this is easier said than done when millions of viewers are watching simultaneously (such as with live events). In the short term, ops teams real-time access to data from the origin to the edge, which includes CDN logs, in order to deliver high-quality viewing experiences. In the long-term, this same data has a wealth of information about how users are interacting with their data (such as when they started and stopped watching and which ads they interacted with).
Data science and business intelligence (BI): Teams looking for deeper insights into their data often need access to long-range historical data for inventory planning, capacity planning, fraud detection, and other mission-critical use cases.

For many enterprises, each of these use cases can quickly become “big data”—extremely large data sets that include terabytes of data on a daily basis and petabytes of historical data. With a tiered storage approach, only a small fraction of that data typically remains hot while the rest is migrated through different tiers until it’s in the cheapest, least accessible form of storage.

With modern solutions that maximize the performance of object storage, the compromise of tiered storage is no longer necessary—enabling more use cases with less operational complexity—all while lowering costs.

How Hydrolix Offers Hot Storage at Cold Storage Costs

Hydrolix is built to solve the problem of runaway data costs while still maintaining the high efficiency and low latency needed for ingesting and querying big data in near real time.

Here’s the TL;DR:

Hydrolix offers the cost-effectiveness of cold storage by writing your data to inexpensive cloud storage and using a patented compression technique to reduce the size of your data by more than 90%. Generally, Hydrolix users get 75% or more savings on total cost of ownership (TCO) compared to solutions they previously used.
Hydrolix decouples storage from compute, giving you the efficiency of hot storage by utilizing massive parallelism, autoscaling stream and query peers, and using advanced querying techniques such as predicate pushdown. You get near-real-time ingest and ad hoc query performance up to 500% faster than other cloud data platforms.

Let’s dig a little deeper into how Hydrolix offers these benefits.

Low-Cost Cloud Storage and High-Density Compression

Instead of using expensive storage solutions like SSDs and storage arrays, Hydrolix data is written to inexpensive cloud storage (generally Amazon, Google, or Azure) in a virtual private cloud (VPC) of your choice. This approach gives you full control over your data at the cost of cold storage. Hydrolix also uses patented high-density compression technology to reduce the footprint of your data.

Stateless Architecture, Parallelism, and Autoscaling for High Efficiency Ingest

To achieve high-efficiency ingest for streaming data, Hydrolix can scale stream peers to transform and write your streaming data using parallelism. Hydrolix uses a unique stateless architecture that decouples storage read and write operations, making it possible to autoscale your stream peers up as needed or even down to zero. This flexibility allows you to manage both high-ingest events by scaling up and off-peak periods by scaling down.

At ingest time, stream peers transform your data on the fly, transforming, indexing, and compressing on a row-by-row basis. Each stream peer writes data to a partition that by default only contains at most sixty seconds of time series data. At first, partitions are small to ensure they are written to storage as quickly as possible so that you can query for actionable insights. As partitions age, they’re merged into larger partitions that use fewer resources. Merging also sorts out-of-order data.

Scalable Query Pools, Predicate Pushdown, and Cataloging for Low Latency Queries

The biggest benefit of hot storage is the ability to query your data on demand and in near real time. With Hydrolix, you can scale the size of your query pools to balance cost considerations versus latency. Large pools with more query peers give you consistent low-latency queries while small pools give you consistent low-cost queries.

Hydrolix leverages a number of advanced techniques to reduce query latency, including a catalog that stores metadata about storage partitions (including minimum and maximum timestamps). When you query your data by time, the query first references the catalog to determine which partitions include that time range, narrowing down the amount of data that needs to be searched and lowering the latency of your queries.

Hydrolix also uses predicate pushdown techniques to improve query performance. By default, all columns are indexed, and Hydrolix partitions include information about where data is stored at the block level. Because all columns are indexed, a query peer can quickly determine which blocks have the requested data and only return those blocks to the query head, making your queries even more efficient.

The end result: hot storage efficiency with cold storage prices.

Come and get your data while it’s hot!

Next Steps

With Hydrolix, all your data is hot—at cold storage prices. If you don’t have a Hydrolix account yet, sign up for a free trial today. With Hydrolix, you get industry-leading data compression, lightning-fast queries, and unmatched scalability for your observability data. Learn more about Hydrolix.