The Benefits of Compression for Big Data

Data compression is a foundational part of streaming data, and we rely on it every day for music, videos, and most content on the internet. Just as importantly, it’s also a critical component of working with big data.

The platforms that can provide novel ways of compressing and improving data truly are cutting-edge technologies, and they can open up use cases that weren’t previously possible.

Hydrolix’s high-density compression maximizes the performance of cost-effective S3-compatible storage for both real-time and historical analytics. The result? A solution that is cost-effective, highly scalable, and offers long-term, always “hot” data storage. None of these benefits would be possible without compression.

Learn about compression in Hydrolix, including how Hydrolix compresses data at ingest time, uses a mix of well-known and proprietary compression techniques, and uses a merge service to further optimize compression over time.

There are plenty of other benefits of compression, too. From a practical perspective, high-density compression can help you dramatically reduce your storage footprint and your compute usage while saving a lot of money.

Meanwhile, from a FinOps perspective, compression should be front and center when it comes to data collection, storage, and analysis. Even minor improvements in compressibility can have a significant impact when you’re working with petabytes of data, allowing you to maximize the value of your data while reducing costs.

This post will cover the benefits of compression for big data and how compression makes solutions like Hydrolix possible.

The Benefits of Compression

Let’s break down some of the benefits of compression—both direct and indirect—for the end user. They extend far beyond just saving money on storage costs.

Smaller storage footprint: Greater compression leads to a smaller storage footprint.
Reduced compute usage: By processing and transferring smaller amounts of data, you don’t need as much compute.
Cost savings: Less compute and a smaller storage footprint lead to reduced costs—but that’s just part of the story. Because compression helps dramatically improve the performance of S3-compatible storage, the storage method itself is also much more cost-effective compared to tightly-coupled storage that uses solid-state drives (SSDs) and storage arrays.
Long-term data retention: With a reduced storage footprint, you can retain more data for longer. Many platforms in verticals like observability struggle with short data retention windows because it is simply too expensive for their customers to retain large volumes of data long-term.
Performant queries on distributed infrastructure: Compression minimizes the amount of data to transfer at query time. While decompression can lead to a small amount of overhead, this is offset by improved transfer efficiency in distributed systems. This makes performant real-time analytics possible even with distributed infrastructure.
Improved scalability: You can scale S3-compatible storage horizontally even for large volumes of data while remaining cost-effective. This is in contrast to tightly-coupled storage solutions that must scale vertically, leading to limitations in their ability to scale as well as high costs for larger volumes of data.
Simplified data management: To manage storage costs with real-time analytics, enterprises typically move data from hot tiers to colder storage tiers that have higher query latency. Many enterprises move data through hot, warm, cold, and frozen tiers. This results in additional management and complexity, especially if you need to continuously fine-tune when data moves between tiers.
Simplified cloud bill: By using S3-compatible storage, all data remains in the same cost-effective tier. There’s no need to navigate multiple tiers of storage and pricing and you can eliminate surprise bills altogether.
Future-proofed for rising volumes of data: The volume of data is rising rapidly for many enterprises, leading to increased costs and higher resource usage. Compression reduces the negative impacts of this trend, especially since it enables horizontally scalable S3-compatible storage for high-performance use cases.
Improved FinOps stance: FinOps is about maximizing the business value of the cloud. This includes not just reducing costs but also extracting more value, such as using long-term retention to maximize the business value of your data.
Improved sustainability: Sustainability is increasingly becoming a core concern for many enterprises, especially when it comes to important issues like energy consumption and climate change. Less storage and compute results in less hardware and less resource consumption.

This doesn’t include one of the most significant benefits for Hydrolix users. A platform like Hydrolix simply wouldn’t be possible without compression.

Why?

Compression helps maximize the performance of S3-compatible storage for real-time analytics. Without compression, S3-compatible storage isn’t performant enough for real-time analytics use cases. And without S3-compatible storage, real-time analytics is too expensive for large volumes of data.

The bottom line: you need compression to make real-time analytics cost-effective.

Why Compression Is So Important for Distributed Architectures

High-density compression is critical for designing high-performance datastores that use distributed architectures. In the case of Hydrolix, compression is a foundational piece for every major subsystem, including ingest, storage, and querying.

Compression Is Fundamental for Large-Scale, Distributed Architectures

TL;DR: Without compression, it wouldn’t be possible to design scalable real-time analytics systems using distributed architecture.

S3-compatible storage systems use cost-effective, distributed hardware to scale horizontally. This makes storage much cheaper than closely-coupled hardware that typically scales vertically.

However, this distributed architecture comes with a tradeoff. Data transfer uses HTTP and must travel further when it’s written, updated, or read from storage. That’s why you can’t simply use S3-compatible storage as a data lake and expect real-time performance, especially as the volume of data scales up. Even if you use a solution like AWS Athena or Google BigQuery, which are powerful analytics tools, you will end up with high costs and subpar performance instead of real-time analytics.

Combined with other techniques like parallelism, high-density compression minimizes the performance tradeoff of distributed architecture. This makes it possible to achieve low-latency reads and writes on your data for real-time analytics.

Let’s take a look at how high-density compression benefits each subsystem in Hydrolix.

Streaming Data at Terabyte Scale

TL;DR: Compressing data before storage leads to more efficient writes and requires fewer compute resources.

A streaming ingest pipeline typically consists of ingesting data from multiple external sources, possibly applying transformations along the way, and then writing that data to storage. In the case of Hydrolix, enterprises typically ingest terabytes of raw, uncompressed data using the Hydrolix streaming API as well as tools like Apache Kafka and AWS Kinesis.

At this level of scale, it’s crucial to write data to S3-compatible storage as quickly as possible while minimizing bottlenecks. One important technique is using parallelism to increase throughput. In the case of Amazon and other major cloud providers, this involves writing data to multiple prefixes in a storage bucket.

It’s also essential to use compression to reduce the amount of data written to S3-compatible storage. As an analogy, if there’s less traffic on the highway, you’re less likely to run into traffic jams or slow traffic. And you may need fewer parallel connections to transfer data, allowing you to minimize the costs of your ingest pipeline.

Low-Latency Queries

TL;DR: Indexes are compressed, too. This allows Hydrolix to index everything and use predicate pushdown to retrieve the smallest amount of data possible from distributed storage.

If you transfer less data, you can do so faster and with fewer compute resources and costs. This is the most obvious benefit of compression on query performance, but once again, it’s only part of the story.

One of the most interesting things about Hydrolix is the fact that all columns are indexed by default. This flies in the face of traditional wisdom, which states that it’s best to be judicious with indexing. This wisdom makes sense for many databases because indexing too many columns can lead to database bloat and poor performance.

However, this is not the case for Hydrolix. There are a number of reasons for this, including Hydrolix’s partitioning strategy and the fact that indexes use a simple list structure, but compression also plays a role.

That’s because indexes are also compressed in Hydrolix. When Hydrolix’s query infrastructure makes a request to a data partition, it can quickly retrieve the partition’s compressed index, determine if the partition has any matching data, and then perform predicate pushdown to retrieve the smallest amount of data possible from the partition.

The following image shows how Hydrolix’s indexing strategy can retrieve narrow ranges of columns, making it more efficient than other columnar databases.

Image shows row-based db on left doing full table scan, columnar db in middle doing full column scans, and Hydrolix on the right doing partial column scans.

In short, compression is a significant reason why all columns can be indexed and why it’s efficient to retrieve a partition’s index for predicate pushdown, leading to low-latency queries.

Storage

TL;DR: Compression doesn’t just reduce your storage footprint. It also makes it possible to use distributed S3-compatible storage for cost-effective real-time analytics and long-term historical analytics.

The most obvious direct benefit of compression is reduced storage costs. If you compress the data in your S3 bucket by 90%, then you’ll save 90% in storage costs.

However, the actual savings are much greater because you need to consider the type of storage (S3-compatible) and the use case (real-time analytics). Without compression, the performance needed to achieve real-time analytics wouldn’t be possible with S3-compatible storage.

This is why legacy real-time analytics platforms (including observability solutions) tend to be so expensive, especially at scale. In order to achieve the performance required for real-time analytics, they were originally built using closely-coupled storage and compute, which relies on expensive solid state drives (SSDs) and large amounts of data kept in local memory. They also often require replicas to ensure high availability, which increases costs even further.

In addition to using cost-effective hardware, S3 and other compatible object storage solutions typically guarantee high availability. That means you don’t need replicas.

The benefits go beyond cost. You can keep data “hot” much longer (15 months by default in Hydrolix), keep more data instead of discarding it, avoid tiered data structures, and open up new use cases that rely on historical data such as business intelligence (BI) and threat hunting.

Compression Increases the Value of Your Data

Hydrolix’s high-density compression reduces the size of data by more than 90%. With log data, it’s common for fields to have low cardinality with many repeated values. Hydrolix users often see compression rates of 20x-50x on these fields.

At 90% compression, a petabyte of data becomes a manageable 100 terabytes. Just as importantly, with Hydrolix, it remains hot for querying in cost-effective, infinitely scalable S3-compatible storage.

This level of compression doesn’t just unlock new degrees of scale. It also allows enterprises to broaden their perspective and see long-term trends in their data.

With use cases like observability, many enterprises focus only on the most recent data. Recent data is extremely valuable for quickly resolving incidents and determining the root cause of issues before they impact customers. But it is short-sighted to only use log data for this purpose.

For many enterprises, it’s not that historical data lacks value—instead, it’s traditionally too expensive to keep for long-term analysis. Teams are forced to think about the short term. With compression and long-term data retention, teams can incorporate better FinOps practices. It’s not just about reducing your cloud bill, but extracting more value out of your data.

If you could keep all your data long-term for analysis, what questions would you ask? And what trends would you look for? Hydrolix can help you answer those questions, allowing you to get more value from your data while paying less for it. And compression is a big part of what makes that possible.