RSS

Building a Log Storage Solution for Global Scale

Learn how Hydrolix achieved 10x revenue growth YoY and another round of Series B funding by building a log solution built for global scale.

Franz Knupfer

Published:

May 22, 2024

6 minute read

This isn’t just another “we got a round of funding” blog, though we’ll lead with that: Hydrolix just raised another $35 million in funding. This is a blog about why we got that funding when many investors and pundits have been downbeat about the tech industry—and how we’re looking at a tenfold increase in revenue year over year at the end of Q2.

And no, it’s not about hustle, or blood, sweat, and tears, or about all the great people working for Hydrolix—though yes, we have hustled, worked hard, and have amazing people at Hydrolix.

It’s about recognizing a critical problem in the tech industry that has a ripple effect across everything from headcount to profitability to budgets. It’s about the cost and volume of incoming data. Many enterprises are dealing with terabyte scale volumes of log data every day, and they have to choose between prohibitively high costs to keep that data or throwing it away. Even worse, sometimes they have to do both—pay too much and discard data.

So how does this issue fit into larger trends in the tech industry? The last few years have seen lost jobs, inflation, and lingering effects from the pandemic. Due to rising costs, there is less budget for headcount, less revenue, less innovation. Enterprises are trying to save more and spend less. They are looking more closely at their budgets and optimizing their cloud spend. They care about FinOps best practices and maximizing the value of their data. They know that data collection, analysis, and storage is a huge budget line item and they’re trying to solve that problem.

There has long been a stereotype of excess in the tech industry—whether that’s elite chefs preparing staff lunches or twelve beers on tap. The real excess these days is nowhere near that fun—enterprises are paying too much for their log data and overprovisioning cloud resources.

In fact, for many enterprises, the cost of storing and analyzing data is second only to headcount. Marty Kagan and Hasan Alayli, co-founders of Hydrolix, experienced this exact issue at their previous company, Cedexis. To provide intelligent load balancing, Cedexis processed more than 15 billion transaction log lines per day—and had to manage massive infrastructure costs as a result. In their case, data was costing nearly as much as headcount, and so they set out to fix the problem.

At Hydrolix, we believe that cost-effective log data at scale is mission critical. It is not just a nice-to-have. It is a utility, a cornerstone of successful, massively distributed systems. And even when the macroeconomic trends are downbeat, utilities need to keep running. Utilities need to be scalable, highly efficient, performant, and cost-effective.

Hydrolix was designed to be all of these things. It is focused on reducing the excessive cost of log-intensive use cases (think one terabyte per day or more) while providing high-performance, real-time analytics, and long-term retention. 

It’s built on the notion that log data has tremendous value, but only if you can afford to keep it and query it. You can measure the cost of data by looking at your monthly cloud bill, but you can’t so easily measure the value and insights lost when data is discarded altogether.

Most cost-cutting strategies come with compromises, so the strategies that give you more value for a lower cost are truly innovative. The Wright brothers’ first flight is what we all remember from the history books, but the advent of cost-effective commercial air travel was far more influential. The same could be said for many innovations from computers to mobile phones. We live in a hyper-connected world not just because these things were invented but because they were commodified. That is exactly what Hydrolix is doing with log data at scale.

Our vision is to be the world’s log storage solution. To forward this vision, Hydrolix uses innovations like cloud-native architecture, decoupled storage, high-density compression, and extreme predicate pushdown to maximize the performance of commodity S3-compatible storage.

When you combine 20x-50x compression with the cost-effectiveness and horizontal scalability of cloud object storage, you can keep all your data as long as you need without discarding it—regardless of whether you are ingesting one or a hundred terabytes per day—and still stay within budget. And Hydrolix gives you sub second query analysis on trillion row data sets regardless of whether the data is a minute or a year old. There are no compromises on performance, or having to discard or tier data, or on costs.

The implications are tremendous. Hydrolix makes many more log use cases economically viable in verticals such as observability, cybersecurity, AdTech, and machine learning. You can threat hunt logs from a year ago without rehydrating data in storage. You can improve model accuracy by training your models on huge datasets. You can understand deeper trends in your infrastructure data and create new features that provide more data and insights to your customers. The possibilities are endless.

We were fortunate to have strong interest in our latest funding round. And this interest can feel validating. Our primary goal, however, is not to fundraise. Our goal is to deliver great services to our clients, grow the business, and uphold our commitments to shareholders and employees. We can only do that if we stay focused on solving the problem that is costing companies jobs, customers, and untold missed opportunities: exorbitant log data costs.

It’s a problem that resonates for both large enterprises and data-driven startups alike. The current macroeconomic trends are simply highlighting the need for better, more cost-efficient log storage solutions at scale, but trends aside, high-efficiency, cost-effective solutions will never go out of style.

Looking Back: Highlights of the Last Twelve Months

Here are just a few highlights of the last year that we’re proud of. They demonstrate our ability to scale for huge events, bring profitability and new features to partners, and work towards our vision of being the log storage solution for the world’s largest and most successful enterprises.

  • In July of 2023, we launched TrafficPeak, a managed observability service, powered by Hydrolix, on Akamai Connected Cloud. Akamai has generated $8 million in TrafficPeak revenue in 9 months.
  • We launched our Powered By Hydrolix program, designed to help enterprises launch new products and features related to observability, CDN monitoring, cybersecurity, AdTech, and other verticals. We’ve got some exciting opportunities in the pipeline that we can’t talk about yet…
  • For the biggest football game of the year, we provided observability at scale for a major media provider. We hit a peak ingestion rate of 11 million rows per second for the customer, with 53 billion records collected and 41 TiB collected in a matter of hours. With Hydrolix’s high-density compression, that data was compressed to a fraction of its original size in storage.
  • We were named Data Breakthrough’s 2024 Cloud Enterprise Data Warehouse Solution of the Year. According to Steve Johansson, managing director of Data Breakthrough, “Hydrolix’s solutions deliver real-time query performance at terabyte scale while dramatically reducing cost for storing and managing log data.”
  • We’ve expanded engineering, customer success, and QE so we can continue to refine our product and build new features. And while we’ll always be focused first and foremost on delivering an amazing product, we know it’s also important to get the word out about what we do and actually sell our products—so we’ve also expanded to include marketing, more sales, and more channel and partner development.

Next Steps

Read the press release and coverage in TechCrunch.

Looking to quickly stand up new revenue streams and products for your customers? Learn about our Powered By Hydrolix partner program.

Interested in maximizing the value of your log data while staying within budget? Try a POC.

Share this post…

Ready to Start?

Cut data retention costs by 75%

Give Hydrolix a try or get in touch with us to learn more

This isn’t just another “we got a round of funding” blog, though we’ll lead with that: Hydrolix just raised another $35 million in funding. This is a blog about why we got that funding when many investors and pundits have been downbeat about the tech industry—and how we’re looking at a tenfold increase in revenue year over year at the end of Q2.

And no, it’s not about hustle, or blood, sweat, and tears, or about all the great people working for Hydrolix—though yes, we have hustled, worked hard, and have amazing people at Hydrolix.

It’s about recognizing a critical problem in the tech industry that has a ripple effect across everything from headcount to profitability to budgets. It’s about the cost and volume of incoming data. Many enterprises are dealing with terabyte scale volumes of log data every day, and they have to choose between prohibitively high costs to keep that data or throwing it away. Even worse, sometimes they have to do both—pay too much and discard data.

So how does this issue fit into larger trends in the tech industry? The last few years have seen lost jobs, inflation, and lingering effects from the pandemic. Due to rising costs, there is less budget for headcount, less revenue, less innovation. Enterprises are trying to save more and spend less. They are looking more closely at their budgets and optimizing their cloud spend. They care about FinOps best practices and maximizing the value of their data. They know that data collection, analysis, and storage is a huge budget line item and they’re trying to solve that problem.

There has long been a stereotype of excess in the tech industry—whether that’s elite chefs preparing staff lunches or twelve beers on tap. The real excess these days is nowhere near that fun—enterprises are paying too much for their log data and overprovisioning cloud resources.

In fact, for many enterprises, the cost of storing and analyzing data is second only to headcount. Marty Kagan and Hasan Alayli, co-founders of Hydrolix, experienced this exact issue at their previous company, Cedexis. To provide intelligent load balancing, Cedexis processed more than 15 billion transaction log lines per day—and had to manage massive infrastructure costs as a result. In their case, data was costing nearly as much as headcount, and so they set out to fix the problem.

At Hydrolix, we believe that cost-effective log data at scale is mission critical. It is not just a nice-to-have. It is a utility, a cornerstone of successful, massively distributed systems. And even when the macroeconomic trends are downbeat, utilities need to keep running. Utilities need to be scalable, highly efficient, performant, and cost-effective.

Hydrolix was designed to be all of these things. It is focused on reducing the excessive cost of log-intensive use cases (think one terabyte per day or more) while providing high-performance, real-time analytics, and long-term retention. 

It’s built on the notion that log data has tremendous value, but only if you can afford to keep it and query it. You can measure the cost of data by looking at your monthly cloud bill, but you can’t so easily measure the value and insights lost when data is discarded altogether.

Most cost-cutting strategies come with compromises, so the strategies that give you more value for a lower cost are truly innovative. The Wright brothers’ first flight is what we all remember from the history books, but the advent of cost-effective commercial air travel was far more influential. The same could be said for many innovations from computers to mobile phones. We live in a hyper-connected world not just because these things were invented but because they were commodified. That is exactly what Hydrolix is doing with log data at scale.

Our vision is to be the world’s log storage solution. To forward this vision, Hydrolix uses innovations like cloud-native architecture, decoupled storage, high-density compression, and extreme predicate pushdown to maximize the performance of commodity S3-compatible storage.

When you combine 20x-50x compression with the cost-effectiveness and horizontal scalability of cloud object storage, you can keep all your data as long as you need without discarding it—regardless of whether you are ingesting one or a hundred terabytes per day—and still stay within budget. And Hydrolix gives you sub second query analysis on trillion row data sets regardless of whether the data is a minute or a year old. There are no compromises on performance, or having to discard or tier data, or on costs.

The implications are tremendous. Hydrolix makes many more log use cases economically viable in verticals such as observability, cybersecurity, AdTech, and machine learning. You can threat hunt logs from a year ago without rehydrating data in storage. You can improve model accuracy by training your models on huge datasets. You can understand deeper trends in your infrastructure data and create new features that provide more data and insights to your customers. The possibilities are endless.

We were fortunate to have strong interest in our latest funding round. And this interest can feel validating. Our primary goal, however, is not to fundraise. Our goal is to deliver great services to our clients, grow the business, and uphold our commitments to shareholders and employees. We can only do that if we stay focused on solving the problem that is costing companies jobs, customers, and untold missed opportunities: exorbitant log data costs.

It’s a problem that resonates for both large enterprises and data-driven startups alike. The current macroeconomic trends are simply highlighting the need for better, more cost-efficient log storage solutions at scale, but trends aside, high-efficiency, cost-effective solutions will never go out of style.

Looking Back: Highlights of the Last Twelve Months

Here are just a few highlights of the last year that we’re proud of. They demonstrate our ability to scale for huge events, bring profitability and new features to partners, and work towards our vision of being the log storage solution for the world’s largest and most successful enterprises.

  • In July of 2023, we launched TrafficPeak, a managed observability service, powered by Hydrolix, on Akamai Connected Cloud. Akamai has generated $8 million in TrafficPeak revenue in 9 months.
  • We launched our Powered By Hydrolix program, designed to help enterprises launch new products and features related to observability, CDN monitoring, cybersecurity, AdTech, and other verticals. We’ve got some exciting opportunities in the pipeline that we can’t talk about yet…
  • For the biggest football game of the year, we provided observability at scale for a major media provider. We hit a peak ingestion rate of 11 million rows per second for the customer, with 53 billion records collected and 41 TiB collected in a matter of hours. With Hydrolix’s high-density compression, that data was compressed to a fraction of its original size in storage.
  • We were named Data Breakthrough’s 2024 Cloud Enterprise Data Warehouse Solution of the Year. According to Steve Johansson, managing director of Data Breakthrough, “Hydrolix’s solutions deliver real-time query performance at terabyte scale while dramatically reducing cost for storing and managing log data.”
  • We’ve expanded engineering, customer success, and QE so we can continue to refine our product and build new features. And while we’ll always be focused first and foremost on delivering an amazing product, we know it’s also important to get the word out about what we do and actually sell our products—so we’ve also expanded to include marketing, more sales, and more channel and partner development.

Next Steps

Read the press release and coverage in TechCrunch.

Looking to quickly stand up new revenue streams and products for your customers? Learn about our Powered By Hydrolix partner program.

Interested in maximizing the value of your log data while staying within budget? Try a POC.