If you work in security operations, you probably know what it’s like to have the data you need but not being able to access it quickly enough for it to be useful. Over time, that constraint changes behavior. Hunts get deprioritized because the queries take too long. Investigations narrow to shorter time windows. Teams adjust their expectations to match what the system can realistically return.
Anton Chuvakin has watched this dynamic play out for more than two decades. In his early SIEM days, vendors stored security events in relational databases, and teams accepted what he calls the “coffee cup query,” a search that should finish in the time it takes to grab coffee. When someone asked for 90 days of data, the answer was different.
“That’s not a coffee cup query,” says Chuvakin. “That’s a vacation query.”
What surprised him most was how normal that limitation felt at the time. Even today, while organizations can retain massive volumes of telemetry at relatively low cost, search performance still determines which workflows remain practical.
“If it takes an hour, a lot of use cases just die,” he says.
Detection, threat hunting, and incident response all depend on the ability to ask new questions and get answers immediately. This guide explores what changes when that speed becomes the baseline and what it takes to keep the security data that matters ready when you need it.
1. Kill “Vacation Queries” Before They Kill Your Use Cases
Slow queries do not just delay analysis. They shape which workflows analysts attempt in the first place.
“Me making a face at five minutes means that if it’s one hour, it’s just completely not applicable,” says Chuvakin. “It’s a full stop. This use case is kaput. It’s not gonna be done.”
He recalls one overloaded system where querying a single day of security data took longer than a day to return. It was “never a design spec,” he says, but it revealed how fragile certain architectures can become. In modern cloud data lakes that optimize for cheap retention, the pattern appears differently. When teams are asked to search nine months of data and the answer is that it would be too slow or too expensive, the workflow quietly disappears.
Best Practices
- Define latency budgets by workflow, not by tool. Detection and interactive hunting are different from audits. Treat them differently.
- Call out your “full stop” thresholds. Anton’s line in the sand is clear: hunting is not about hour-long queries. You need fast pivots, or you will not pivot at all.
- Test the uncomfortable timeframe. Run your common hunts and compromise checks over 30, 90, and 180 days. If it becomes a vacation query, you have your answer—your platform is quietly killing your ability to get your work done.
2. Treat Compromise Assessment Like a Speed Drill
When you suspect that a compromise has occurred, the goal is not a single definitive answer. It is momentum.
“I want to run a massive number of queries for many type of scenarios that may indicate a compromise,” says Chuvakin. He wants immediate feedback so that any signal can be escalated and investigated without delay.
“I don’t want to have 10 coffee cup queries,” Chuvakin says. The value is not in any one query. It is in being able to pivot quickly, adjust your hypothesis, and test again.
“Google doesn’t tell you, ‘Hey, you’re looking for a fairly complicated website. It’ll take a while,’” Chuvakin says. “You just get a result immediately.”
He wants security telemetry to operate at that standard.
Best Practices
- Write a compromise checklist that assumes fast search. Make it a set of queries you can run quickly, not a set of steps that require people to manually inspect systems or do other time-consuming work.
- Optimize for parallel thinking. The value is not in one query. It is running many hypotheses quickly and pivoting immediately.
- Make “Google speed” the expectation. Consumer search works instantly. Internal security workflows should not tolerate multi-hour waits, either.
3. Redefine “Hot vs Cold” as a Use Case Decision, Not a Technical Doctrine
Chuvakin jokes that the hot versus cold distinction should have stayed in the 1990s. In his ideal world, if data is worth keeping for real use cases, it should be available.
“If I want to keep a terabyte, a petabyte, or whatever other byte, somebody is paying for hard drives,” says Chuvakin.
The question is not whether storage exists. It is whether the workflow justifies the cost of speed. “I don’t want to debate cold versus hot,” Chuvakin says. “Just give me the data. I want to search. I want it now.” Rather than starting with infrastructure, he starts with use cases. A quarterly compliance report may tolerate delay. Detection and priority incident response can’t.
Best Practices
- Label data by workflow.
- Hot: Detection, hunting, priority incident response need hot storage.
- Cold: Auxiliary analysis, overnight ML, and batch reporting where tomorrow morning is fine can be kept in cold storage.
- Be explicit about what you will tolerate. If you are okay with data showing up tomorrow morning, it belongs in cheaper storage. If not, it needs to stay hot.
- Avoid the false binary. Some scenarios are not about better storage at all. They are about streaming. Certain use cases need near real-time stream processing, which solves the problem differently than searching stored data.
4. Stop Duct-Taping Five Storage Layers Together
Performance and cost are visible metrics. Architectural resilience is not. Systems stitched together with APIs, “duct tape,” and “hope” may satisfy budget requirements and pass initial tests, but the stress test comes later.
“I want this to work under pressure,” says Chuvakin. “I want it to work under the attack. I want it to work in a stressful situation of incident response.” Each redirect between storage tiers introduces another potential failure point at the exact moment reliability matters most.
Best Practices
- Count the number of “where is the data” steps in your SOC. Every handoff and redirect adds delay.
- List your failure modes. What happens if a provider discontinues one layer? What if one layer is misconfigured?
- Optimize for “works under pressure.” The best architecture is not the one that demos well. It is the one that holds up when you need it most.
5. Measure Success by Use Cases, Not Vanity Metrics
When asked how to judge success, Chuvakin goes straight to use cases. “Am I getting the value where speed and cost and quality of data are in the right balance?” says Chuvakin. For a quarterly compliance report, he does not need hot data. If it takes days and saves money, that may be acceptable. For hunting, detection, and priority incident response, he wants speed, correct scope, and appropriate timeframe coverage.
Put this into practice by tracking:
- Time-to-answer for top detection and hunting workflows measured on realistic history windows.
- Pivot depth meaning how many follow-up queries an analyst can run before abandoning a thread.
- Coverage retention meaning how far back you can search at interactive speed for your most important use cases.
6. If You Can’t Have Hot Data Everywhere, Fight for Non-Negotiable Use Cases
Budget and change management do not always cooperate. Chuvakin’s advice is pragmatic.
“You save money, but you lose the capability,” says Chuvakin. Identify the smallest number of use cases that absolutely can’t function with slow access, and protect those first.
He also notes that some use cases should shift from stored analytics to streaming analytics. In those cases, you are not solving the problem with faster storage. You are solving it with near real-time stream processing. The goal is not perfection. It is preserving the workflows that define your security posture.
Best Practices
- Build a short “non-negotiables” list. Identify the smallest set of use cases where delay is unacceptable and prioritize faster access for those first.
- Make the trade-offs explicit. Write down what you are willing to let run slower elsewhere, because you save money but you lose capability.
- Choose the right speed mechanism. Separate faster storage needs from true real-time needs and use streaming where the use case demands it.
Final Takeaway
When query speed degrades, the surface area of your security program shrinks. Threat hunting gets slower and harder to do, and detection windows narrow. As a result, analysts may give up on some queries and hypotheses altogether. Chuvakin has seen teams quietly adjust to system limitations instead of challenging them. Slow search does not just introduce delay. It reshapes what is possible.
He also warns against over-engineered systems that rely on “duct tape” and “hope.” They may satisfy budget goals on paper, but the real measure is whether they function during an attack or a high-stress incident.
The standard is simple: your architecture must hold up under pressure. That reframes the conversation. It’s not just about retention or storage cost. It is about whether your most important workflows remain viable when the stakes are high.
With Hydrolix, security teams can:
- Keep and analyze security log data at any time, whether it’s a minute old or a year old.
- Retain data long-term in object storage with high-density compression (typically 10x-50x) for a reduced storage footprint, all while keeping data hot for queries.
- Support more log-driven security use cases by handling streaming ingest, data transformation, and compression.
- Shift from “hot vs cold tradeoffs” to always-ready answers by keeping more data hot for security and incident response workflows.
- Make long-retention log analytics practical with a platform built for always-hot log analytics and sub-second queries.
Next Steps
If “vacation queries” are breaking your workflows and you need to keep more security telemetry hot, Hydrolix can show you what log analytics on both real-time and historical data looks like in practice. Request a Hydrolix demo.
