A couple of years ago, ingesting ten terabytes of log data per day was considered a lot. But for many enterprises, this volume of data has become the new normal. For live events, companies can ingest hundreds of terabytes in just a few hours. Some companies ingest petabytes per day, which is an incredible amount of log data.
With dispersed microservices and products such as Kinesis and AWS Lambda, all of which generate logs, coupled with the rapid adoption of cloud-native everything and AI, data is everywhere. The monolithic applications of yesterday are no longer. Parts of applications run on different servers and they’re spread out globally.
Suddenly, one application has become 50 services. Each one has its own log output and status. And while having more data and a cloud-native infrastructure is beneficial for speed, visibility, and reliability, it can also cause pain. In addition to being tougher to monitor and manage data, legacy observability services weren’t built for this level of scale and they are expensive. The conundrum leads to data being sampled or discarded (which comes at a cost).
The pain of too much data on legacy systems is starting to outweigh the expense of changing observability platforms. Enterprises need scalability and flexibility, at a price point that doesn’t break the budget.
Hydrolix and AWS
That’s where Hydrolix comes in. Recently, we solidified our partnership with AWS to offer managed observability for multi-CDN, origin, and AWS edge services such as CloudFront, WAF, and Elemental. We are already a CloudFront Service Ready partner and joined ISV Accelerate, AWS’s main co-selling program.
Hydrolix was built to ingest, retain, and analyze massive amounts of log data—at least a terabyte a month—at a cost-effective price point. We retain data for at least 15 months, about a year longer than other observability providers, while keeping all data hot for real-time querying at 75% less the cost of legacy solutions. Hydrolix uses decoupled object storage, which is highly scalable by design. Most legacy observability services use tightly-coupled storage, and companies have to determine ahead of time how many virtual machines and disks they need. But when it comes to logs, that number can be hard to predict. If something breaks, the log volume skyrockets, and if the service doesn’t scale to handle the increase, then logs aren’t ingested quickly enough.
In contrast, Hydrolix can autoscale to handle high volumes of data—or even scale down to zero to save on costs. Because Hydrolix is designed to maximize the query performance of object storage, users get subsecond “hot” query latency for all data regardless of age.
Hydrolix is solving the financial and scalability pain points that come with legacy observability solutions. Companies simply give us an S3 endpoint and we can do the rest. Whether they ingest 100 terabytes or one terabyte a day, it doesn’t matter. Hydrolix has handled major events that ingested hundreds of terabytes over the course of a few hours. That volume of log data would take days to scale with a legacy provider. Even with users and dashboards running tens of thousands of queries against that data, the average query took less than a second to complete.
Observability isn’t the only use case for Hydrolix. Imagine being able to keep enough data to create more accurate machine learning models, or AI agents using historical log data to tune their performance. With log data coming in in real time, companies can tune their models so that they flag anomalous behavior and other issues. The quality of an AI agent is only as good as the data fed into it and the ability to query that data effectively.
Enterprises are increasingly recognizing that whole new opportunities open up when they have true access to their log data.
Next Steps
- Learn more about Hydrolix for AWS.
- Try Hydrolix for AWS free through the AWS Marketplace.
