6 Ways to Cut Your Splunk Bill

Your Splunk bill keeps going up. Meanwhile, your CFO wants to bring costs back down. This scenario is playing out for cybersecurity, observability, and ops teams across the globe. Splunk is a popular, versatile tool, but it’s expensive. A 600GB/day license for Splunk can cost enterprises more than one million dollars per year, and many enterprises are dealing with terabytes of log data per day.

For Splunk users, the scenario in the following comic (xkcd-style) might feel all too familiar.

Leaders and teams shouldn’t have to choose between observability and the budget, but too often, that’s what happens.

So how do you keep Splunk without losing visibility into your data? Here are six strategies for cutting your Splunk bill by 50% or more. Some involve making sacrifices in terms of the data you keep (sampling), but others (such as tiered observability) allow you to keep all your data while controlling costs.

TL;DR: With tiered observability and data federation, you can keep all your data long-term, reduce costs for observability and cybersecurity, and continue to visualize your data in Splunk.

1. Filter Logs

Are you sending DEBUG, TRACE, and INFO logs to Splunk? Or are there logs from overly chatty services that you might not need to analyze in the first place? With log filtering, you can filter out some of your logs instead of sending them to Splunk. For example, you can use a telemetry pipeline tool like Cribl Stream to filter logs. Some users of Cribl Stream successfully filter 25-50% of their logs, dramatically reducing their Splunk bill.

This approach does come with some gotchas. For example, it may be tempting to filter logs of successful transactions. An example might be HTTP request logs to static assets that have 200 OK status codes. After all, if these transactions were successful, why retain them? But they often include other valuable data. For example, what if a request took an unusually long time to complete, which could indicate a poor quality of experience (QoE) for users?

While filtering logs can reduce your Splunk bill, processing data in a telemetry pipeline comes with an extra cost. In the case of Cribl Stream, for example, an enterprise account costs $.32/GB (with additional infrastructure costs on top), a high price to pay for throwing data away.

Quick win: Audit your top five sources of log data. Are any of them emitting DEBUG, TRACE, and INFO logs, and can you safely filter them?

2. Audit Services with Verbose Logs

If you’re filtering logs that you never intend to keep, why send them elsewhere in the first place? While Cribl Stream makes it easy to filter verbose logs, the cost of using it quickly adds up, especially if you’re ultimately discarding the extra data. If this is the case, you can save money by auditing logs at the source and cutting down on verbosity and frequency.

For example, for services using Python, there are several approaches you could take to suppress lower log levels such as DEBUG. You could increase the minimum log level across an environment:

logging.basicConfig(level = logging.WARNING)

This would suppress DEBUG and INFO logs. You can also suppress logs for specific modules or environments, set up conditional logging, or even use filters at the source. In general, this is going to be a much more cost-effective approach than filtering logs through a telemetry pipeline that costs extra to process your data.

Quick win: Same as for #1. Audit your top five sources of log data, or, if you’re using a service like Cribl, see where you’re spending the most money on processing and filtering data. Can you filter or otherwise reduce that data at the source and avoid sending it through a telemetry pipeline that comes with extra costs?

3. Reduce Dimensionality by Dropping Columns

Dimensionality refers to the “width” (or total number of attributes) of a log line. Logs can be high dimensionality (hundreds of columns or more), and if there are some attributes that are never being used for analytics, then you’re paying extra for those columns.

To reduce costs, you can filter your log data by dropping columns instead of entire log lines. This can be beneficial in terms of reducing security risks, too. For example, you should audit your production logs to ensure that development-specific columns aren’t being captured. Development logs can potentially expose source code, and in a production environment, this increases the risk that malicious actors can discover vulnerabilities or steal intellectual property.

You may also want to drop columns that expose personally identifiable information (PII). This reduces potential security and compliance risks and saves on the bottom line.

In Splunk, you can see all the columns in a table using pipes with the fieldsummary command. Add the following SPL pipe to the query to get a list of all the fields included in a table.

… | fieldsummary | table field 

To just get a count, you can add this pipe:

| stats count

If the count is high, or fieldsummary reveals fields that aren’t being used for analytics, those could be good candidates for dropping. Just be careful that you don’t inadvertently drop columns that you might need later.

Alternatively, if you are storing the data in another database, you can query the table schema to get information about the total number of columns. Some tools (such as Hydrolix) provide flexible schema, so different log lines can have varying numbers of columns.

Interesting fact: Modern observability is increasingly adopting observability 2.0: arbitrarily wide logs. This means that logs can have varying numbers of attributes, and that observability platforms should include flexible schema and the capacity to analyze logs of varying dimensionality. Dropping columns can save on Splunk costs, but expect to see more observability platforms with the ability to better manage the potential costs of high dimensionality. Databases that use columnar formats can efficiently compress columns for cost-effective storage and can also target individual columns in queries, making compute more cost-effective.

4. Sample Logs

With sampling, enterprises can retain just a percentage of incoming data. There are many approaches to sampling. The simplest approach is random sampling, which can lead to data gaps. A better approach is to sample based on log characteristics using stratified sampling or cluster sampling.

• Random sampling: This approach randomly selects which logs to retain. It’s simple to implement and doesn’t involve pre-processing logs to determine how they should be sampled. But it can lead to significant gaps in data.
• Stratified sampling: This smarter approach allows you to select different sampling rates for different strata of logs. For instance, you might want to retain 100% of logs that have a log level of CRITICAL but a smaller percentage that have a log level of WARNING. Once again, you need to be careful not to create gaps in data. For instance, if you were to set a lower sampling rate for HTTP request logs that have a status code of 200 OK, you might miss other issues such as high request latency or malicious behavior.
• Cluster sampling: Logs can also be sampled based on other types of groupings, such as by server or geographic location. This technique can be helpful when log data is logically clustered. Once again, data gaps can be an issue, and you may end up with unequal representations of data across clusters.

Random sampling takes less processing because it’s head-based. There’s no need for a service to read the log to see if it meets sampling criteria before it’s discarded. Smart sampling takes more processing because it’s often based on characteristics of the log line. Sampling is particularly common with distributed traces, which trace requests through systems.

Stream processing solutions like Cribl can sample data before sending the rest to Splunk, allowing you to control the amount of data that Splunk ingests.

Key gotcha: With sampling, you’re guaranteed to have gaps in your data. That’s just how sampling works. If you need to keep data for cybersecurity or compliance reasons, sampling is not the right strategy.

5. Use Tiered Observability

With tiered observability, log data is categorized into tiers. Typically, the tiers are gold, silver, and bronze. The most valuable gold data is stored in a premium observability solution such as Splunk. Silver tier data is stored in a less expensive platform, such as a log analytics platform that has fewer features. Bronze tier data can be stored in a data lake for increased cost efficiency.

The following chart, from this post on maximizing the benefits of tiered observability, shows the breakdown.

Tiered observability involves some planning and consideration. You need to think carefully about the value of your log data and how it should be separated. If you’re not careful, you can end up with multiple silos for your log data and headaches for the teams that need to analyze it. Only the most valuable data stays in Splunk (typically 10% or less), which can also mean compromises in terms of the tools that your teams use, especially if most data is stored in a solution with much more limited features.

Pro tip: By combining tiered observability with data federation, you can store more data in cost-effective solutions while using Splunk to federate data. You get major cost savings while your teams can continue to use Splunk for analytics.

6. Federate Log Data with Splunk

Data federations allow you to query data from multiple different sources in a single interface. In this scenario, you can use Splunk as the query interface while retaining data in cost-effective storage.

The most effective way to do this is by using a log analytics solution that has an integration with Splunk. Hydrolix Search for Splunk, which is available in Splunkbase, allows you to store your data in Hydrolix while querying it in Splunk. Enterprises using Hydrolix to ingest and store data typically save 6x or more when keeping data in Hydrolix instead of Splunk. The cost of retaining data in Hydrolix is more cost-effective than ingesting and filtering data through Cribl Stream, and you get to keep all of it without compromising on your budget. This approach combines data federation with tiered observability to give you the best of both worlds: real-time analytics and cost-effective data retention for all of your data, even when you’re ingesting high volumes of logs on a daily basis.

Alternatively, Splunk allows you to federate Splunk indexes stored in different locations. One option is to combine federated Splunk with AWS Glue so that you can store your Splunk indexes in cost-effective S3 storage. You still do need to ingest the data into Splunk to index it, though, and this approach is really for extending Splunk retention at a more cost-effective rate. The other downside is that query performance won’t be optimal, making this approach suitable for basic compliance but not active analytics use cases. In general, solutions that are designed for high performance on object storage such as Hydrolix will be a better storage approach for storing data long-term.

The following image shows an example of analyzing SIEM data stored in Hydrolix in a Splunk dashboard with SPL.

Next step: Read more about analyzing Hydrolix data in Splunk, complete with use cases, queries, and dashboard screenshots.

The Bottom Line: You Don’t Have to Compromise

Traditionally, data reduction has been the primary way to cut Splunk costs. Whether that’s sampling, filtering, or reducing dimensionality, the end result is the same: discarded data.

In some scenarios, discarding or filtering can be the best approach, but if you’re discarding data that you’d rather keep just to cut costs, you shouldn’t be forced to make that compromise. Instead, you can combine tiered observability with data federation so you can keep all your data, analyze it and Splunk, and save money, too.

The No-Compromises Solution

• Keep recent, high-value data in Splunk for immediate analysis. This is the gold tier of tiered observability, and should typically be less than 10% of total data volume.
• Stream other logs to Hydrolix for cost reduction and long-term retention. Hydrolix can manage the silver and bronze tiers, which should be about 90% of your data.
• Use Hydrolix Search for Splunk to query your Hydrolix data in Splunk.
• Maintain existing workflows while dramatically reducing costs. Enterprises typically can reduce costs 6x or more by using Hydrolix instead of Splunk for data.

Reduce Your Splunk Bill Starting Today

For Splunk customers, every day without a cost reduction strategy can potentially lead to thousands of dollars in unnecessary costs. For enterprises dealing with large volumes of data, that adds up to hundreds of thousands or even millions of dollars every year. That money could be better spent increasing margins, building your teams, and balancing budgets.

Reducing costs can seem overwhelming at first, especially as log volumes continue to rise. Start by auditing services with verbose logs. Can you cut down on the noise, or do you need to keep it all? Once you’ve identified the biggest sources of log data, you can determine whether filtering and sampling or tiered observability is the right strategy for you. Or combine both approaches as needed.

Check out our pricing calculator to see how much you can save by using Hydrolix instead of Splunk for the majority of your data. With a free managed trial, you can start ingesting data into Hydrolix today. Ingest logs into both Splunk and Hydrolix, compare costs, and get hands on with the Hydrolix Connector for Splunk to see how tiered observability can slash your Splunk bills.