RSS

Splunk and Other Observability Vendors Cost Too Much

Uncertainty around the Splunk acquisition underscores a key problem for customers—observability costs across the industry are too high.

Franz Knupfer

Published:

Apr 02, 2024

6 minute read

Table of Contents

It’s a done deal—Cisco’s acquisition of Splunk is complete. But there’s still a lot of uncertainty around what will happen with Splunk going forward. Will prices go up even more? Will feature innovation slow? How about the risk of increased vendor lock-in?

The writing is already on the wall in terms of increasing costs. Just look at Splunk’s pricing policy for renewing customers. “To align with industry standards, Splunk has adopted a Standard Uplift policy for renewals to incorporate additional value driven by product enhancements and increased cost of services.”

This “Standard Uplift” amounts to 9% for a 1-year renewal, 7% per year for a 2-year renewal, or 5% for a 3-year renewal. How much more “uplifting” news can Splunk customers handle before costs get too high? Yes, you can lock in lower increases—if you agree to vendor lock-in.

Splunk has built a versatile product that many users rely on. Over 20+ years, Splunk has developed everything from a powerful query language for search and analysis to a well-developed UI. But under the hood, not everything can easily be “ripped and replaced” or “lifted and shifted.” For all the “additional value driven by product enhancements” that gets trumpeted in product launches and announcements, there is plenty of technical debt that isn’t talked about, but is still rolled into the “increased cost of services” nonetheless.

So let’s talk about the (very big) elephant in the room: all that data coming in. And let’s talk about how much it costs to store all that data long-term—or even short-term, because many enterprises have no choice but to quickly discard data or move it to cold storage to reduce costs. Those yearly price hikes are in addition to rapidly increasing volumes of data. 

According to a survey of DevOps and SRE professionals, log data has increased by 5x over a three year period. 51% deal with unexpected overages and cost spikes on a monthly basis. And 36% of companies are ingesting more than a terabyte of data on a daily basis.

Splunk is under the microscope because of Cisco’s eye-opening acquisition, but they are not alone in dealing with rising data volumes and costs. It’s a huge problem across the observability industry. From Elastic’s high costs to the infamous story of a Datadog customer getting a $65 million bill, platforms are struggling to keep costs down, leaving customers with huge observability bills. It’s why so many vendors spin the problem of data loss into supposed “best practices” like sampling and limited data retention windows.

Learn about four common observability anti-practices that are typically described as “best” practices—and how you can avoid them.

Throwing away data isn’t the right answer for most use cases. Instead, observability platforms—as well as other platforms that rely on big data—need to transform their approach to data ingest, storage, and analysis from the ground up. Too many platforms are built on legacy storage architectures that are tightly coupled, expensive, and difficult to scale. When scaling becomes too expensive, costs are passed on to platform users—or users must throw away data instead.

Marty Kagan, CEO and co-founder here at Hydrolix, wrote about the Splunk acquisition and the need for a new approach to data storage for AI use cases in Inside Big Data: “Architectures are emerging that decouple storage, allowing compute and storage to scale independently, and index that data so that it can be searched quickly. This provides solid-state drive-like query performance at near object storage prices.”

These kinds of architectures can and must power the observability platforms of the future. By building on cost-effective object storage, platforms can offer more features at a lower cost while still maintaining healthy margins. And the need for affordable storage at scale will become even more critical as log volumes continue to increase. For example, AI is both data-hungry (for training models) and also generates vast amounts of log data. To maximize the accuracy and performance of AI models, enterprises need to keep all of their data.

Transforming the Economics of Log Management with Next-Gen Cloud Data Platforms for Observability Use Cases

Learn why most observability platforms cost too much, and how modern cloud data platforms must push the limits of object storage to offer long-term, cost-effective log data.

Marty experienced the problems of costly data at scale firsthand at his previous company. As he wrote: “At our last company, my co-founder and I saw our data volumes grow to over 15 billion transaction logs a day. As our infrastructure costs went through the roof, we faced the unhappy choice of throwing data away, paying through the nose, or accepting poor performance. Everywhere we looked in the industry, we saw organizations burning too much money on their data and being forced to make similar tradeoffs, cutting short the opportunity to invest in long-term data retention and deep learning.”

Hydrolix was built to solve this problem.

Hydrolix is designed to allow you to retain long-term “hot” data. Customers typically save 75% or more over their previous solution while getting 15-month hot data retention windows. Meanwhile, the industry standard is 30 days to a few months.

Hydrolix is a columnar, append-only datastore optimized for time-based event data and designed to ingest data in real time at terabyte scale. It takes advantage of the cost benefits of object storage, combining decoupled architecture with independently scalable ingest and query subsystems. Hydrolix runs on Kubernetes clusters in an enterprise’s virtual private cloud (VPC), making it an ideal solution for multi-tenancy. You can learn more about our approach to maximizing object storage in our Powering Big Data with Next-Gen Cloud Data Platforms whitepaper. 

Observability platforms and SaaS providers dealing with high data costs and legacy data solutions can build on or partner with Hydrolix to offer customers affordable data at scale while maintaining healthy profit margins. By doing so, enterprises can stand up new features and solutions much faster than building from scratch. No need to “lift and shift” or “rip and replace” —rather, you can “augment and accelerate.”

As a case in point, we’ve partnered with Akamai to offer TrafficPeak, a managed observability platform that offers 15 months of hot data retention to customers along with streaming ingest at terabyte scale and sub-second query latency. The solution was ready for Akamai customers in a matter of months, and now provides observability to some of the biggest companies in the world.

Cisco’s acquisition of Splunk is a done deal, but customers may look for new solutions if costs get too high. Some customers are already reaching that inflection point. Throwing data away isn’t a best practice or the right answer when it comes to cutting costs and staying within budget. The problem of data costs at scale is an imminently solvable problem. Both the pioneers of the past and the data platforms of the future can work together to make long-term retention and analysis of big data affordable.

Next Steps

Read Transforming the Economics of Log Management to learn about the issues facing many of today’s observability platforms, and how next-generation cloud data platforms must maximize the benefits of object storage to make log data cost-effective.

Learn more about how Hydrolix offers cost-effective data at terabyte scale and contact us for a POC.

Share this post…

Ready to Start?

Cut data retention costs by 75%

Give Hydrolix a try or get in touch with us to learn more

It’s a done deal—Cisco’s acquisition of Splunk is complete. But there’s still a lot of uncertainty around what will happen with Splunk going forward. Will prices go up even more? Will feature innovation slow? How about the risk of increased vendor lock-in?

The writing is already on the wall in terms of increasing costs. Just look at Splunk’s pricing policy for renewing customers. “To align with industry standards, Splunk has adopted a Standard Uplift policy for renewals to incorporate additional value driven by product enhancements and increased cost of services.”

This “Standard Uplift” amounts to 9% for a 1-year renewal, 7% per year for a 2-year renewal, or 5% for a 3-year renewal. How much more “uplifting” news can Splunk customers handle before costs get too high? Yes, you can lock in lower increases—if you agree to vendor lock-in.

Splunk has built a versatile product that many users rely on. Over 20+ years, Splunk has developed everything from a powerful query language for search and analysis to a well-developed UI. But under the hood, not everything can easily be “ripped and replaced” or “lifted and shifted.” For all the “additional value driven by product enhancements” that gets trumpeted in product launches and announcements, there is plenty of technical debt that isn’t talked about, but is still rolled into the “increased cost of services” nonetheless.

So let’s talk about the (very big) elephant in the room: all that data coming in. And let’s talk about how much it costs to store all that data long-term—or even short-term, because many enterprises have no choice but to quickly discard data or move it to cold storage to reduce costs. Those yearly price hikes are in addition to rapidly increasing volumes of data. 

According to a survey of DevOps and SRE professionals, log data has increased by 5x over a three year period. 51% deal with unexpected overages and cost spikes on a monthly basis. And 36% of companies are ingesting more than a terabyte of data on a daily basis.

Splunk is under the microscope because of Cisco’s eye-opening acquisition, but they are not alone in dealing with rising data volumes and costs. It’s a huge problem across the observability industry. From Elastic’s high costs to the infamous story of a Datadog customer getting a $65 million bill, platforms are struggling to keep costs down, leaving customers with huge observability bills. It’s why so many vendors spin the problem of data loss into supposed “best practices” like sampling and limited data retention windows.

Learn about four common observability anti-practices that are typically described as “best” practices—and how you can avoid them.

Throwing away data isn’t the right answer for most use cases. Instead, observability platforms—as well as other platforms that rely on big data—need to transform their approach to data ingest, storage, and analysis from the ground up. Too many platforms are built on legacy storage architectures that are tightly coupled, expensive, and difficult to scale. When scaling becomes too expensive, costs are passed on to platform users—or users must throw away data instead.

Marty Kagan, CEO and co-founder here at Hydrolix, wrote about the Splunk acquisition and the need for a new approach to data storage for AI use cases in Inside Big Data: “Architectures are emerging that decouple storage, allowing compute and storage to scale independently, and index that data so that it can be searched quickly. This provides solid-state drive-like query performance at near object storage prices.”

These kinds of architectures can and must power the observability platforms of the future. By building on cost-effective object storage, platforms can offer more features at a lower cost while still maintaining healthy margins. And the need for affordable storage at scale will become even more critical as log volumes continue to increase. For example, AI is both data-hungry (for training models) and also generates vast amounts of log data. To maximize the accuracy and performance of AI models, enterprises need to keep all of their data.

Transforming the Economics of Log Management with Next-Gen Cloud Data Platforms for Observability Use Cases

Learn why most observability platforms cost too much, and how modern cloud data platforms must push the limits of object storage to offer long-term, cost-effective log data.

Marty experienced the problems of costly data at scale firsthand at his previous company. As he wrote: “At our last company, my co-founder and I saw our data volumes grow to over 15 billion transaction logs a day. As our infrastructure costs went through the roof, we faced the unhappy choice of throwing data away, paying through the nose, or accepting poor performance. Everywhere we looked in the industry, we saw organizations burning too much money on their data and being forced to make similar tradeoffs, cutting short the opportunity to invest in long-term data retention and deep learning.”

Hydrolix was built to solve this problem.

Hydrolix is designed to allow you to retain long-term “hot” data. Customers typically save 75% or more over their previous solution while getting 15-month hot data retention windows. Meanwhile, the industry standard is 30 days to a few months.

Hydrolix is a columnar, append-only datastore optimized for time-based event data and designed to ingest data in real time at terabyte scale. It takes advantage of the cost benefits of object storage, combining decoupled architecture with independently scalable ingest and query subsystems. Hydrolix runs on Kubernetes clusters in an enterprise’s virtual private cloud (VPC), making it an ideal solution for multi-tenancy. You can learn more about our approach to maximizing object storage in our Powering Big Data with Next-Gen Cloud Data Platforms whitepaper. 

Observability platforms and SaaS providers dealing with high data costs and legacy data solutions can build on or partner with Hydrolix to offer customers affordable data at scale while maintaining healthy profit margins. By doing so, enterprises can stand up new features and solutions much faster than building from scratch. No need to “lift and shift” or “rip and replace” —rather, you can “augment and accelerate.”

As a case in point, we’ve partnered with Akamai to offer TrafficPeak, a managed observability platform that offers 15 months of hot data retention to customers along with streaming ingest at terabyte scale and sub-second query latency. The solution was ready for Akamai customers in a matter of months, and now provides observability to some of the biggest companies in the world.

Cisco’s acquisition of Splunk is a done deal, but customers may look for new solutions if costs get too high. Some customers are already reaching that inflection point. Throwing data away isn’t a best practice or the right answer when it comes to cutting costs and staying within budget. The problem of data costs at scale is an imminently solvable problem. Both the pioneers of the past and the data platforms of the future can work together to make long-term retention and analysis of big data affordable.

Next Steps

Read Transforming the Economics of Log Management to learn about the issues facing many of today’s observability platforms, and how next-generation cloud data platforms must maximize the benefits of object storage to make log data cost-effective.

Learn more about how Hydrolix offers cost-effective data at terabyte scale and contact us for a POC.