Unlocking Threat-Hunting Insights From High-Volume Network Logs

When it comes to cybersecurity, gathering and analyzing data is more crucial than ever to staying ahead of malicious actors. Data from sources like DNS query logs, netflow logs, CDN logs, and WAF logs often contains valuable information that can prevent or mitigate threats.

Unfortunately, many companies take the path of least resistance by keeping this data for a very short time or storing high-level summaries that lack granular insights. The data is then discarded or moved to long-term cold storage where it becomes practically unusable—essentially going dark.

This wealth of network and application data can help organizations improve their threat hunting protocols and operations. It can also help teams understand what went wrong when breaches or attacks occur. As advances are made in machine learning and more enterprises adopt optimized, cost-effective data storage solutions, it will become increasingly common to use this longer-term data for security and threat-hunting purposes.

The Rise of Slow and Cunning Cyber Threats

Short-term data retention windows don’t just create gaps in visibility. They also lead to increased vulnerability and new attack vectors. Sophisticated attackers often take a “low and slow” approach that’s much harder to detect. Low and slow attacks are designed to avoid detection by standard security protocols. The breadcrumbs these attacks leave behind quickly disappear when enterprises have short data retention windows. Similarly, advanced persistent threats (APTs) may involve coordination and data extraction over long periods, slipping under the radar of conventional monitoring.

These more sophisticated and patient attacks are on the rise, and organizations need long-term data analysis to detect and mitigate them.

To address these challenges and opportunities, let’s explore how organizations can unlock security benefits and use advanced techniques like threat hunting on common sources of high-volume logs, including:

DNS queries
Netflow logs
CDN logs
WAF logs

Each log type gives unique insights into network activity, and combining them can enhance security intelligence.

What Is Threat Hunting?

Threat hunting is a proactive security approach where security practitioners actively search for signs of malicious activity or threats within a network or system before they trigger alerts or lead to issues like major breaches.

Threat hunters assume that malicious actors are already inside networks and systems and use tools like manual analysis of transaction logs, hypothesis-driven investigation, and behavioral analytics to uncover threats. These are just a few examples of breadcrumbs that threat hunters might look for:

Indicators of compromise (IoCs) such as suspicious IP addresses, malicious domains, and unusual protocols and ports.
Anomalous activity such as large data transfers or login times when employees aren’t typically working.

With threat hunting, security professionals can:

Detect advanced persistent threats (APTs) that evade traditional defenses.
Identify insider threats such as misuse of privileges by insiders.
Uncover zero-day exploits or novel attack methods.
Improve overall detection capabilities by inputting their findings into security tools.

Many of the logs that threat hunters must sift through are high volume, and successfully tracking down threats involves looking at logs over an extended time span of months or even years.

DNS Query Logs

DNS logs track domain name resolutions, translating human-readable domain names to IP addresses. DNS is basically the telephone directory of the internet. DNS queries and responses are the first processes involved in serving up internet content and, as such, they’re a frequent attack vector. Also, because the DNS protocol is so integral to the internet, DNS traffic moves more easily through firewalls, often with less scrutiny than other types of requests such as HTTPS, FTP, and SMTP.

Types of DNS-related attacks can include DNS hijacking, DNS tunneling, DNS cache poisoning, command-and-control communication, and various types of denial-of-service attacks. According to an EfficientIP report for 2023, the average cost of a DNS attack was $1,000,000, and 90% of organizations were subject to a DNS attack.

How much DNS data an organization has access to for analysis depends on a number of factors, including the industry type, whether they operate their own DNS servers, and more. For example, an average company may have granular DNS query logs at the level of specific computers and users. A managed service provider (MSP) might see only DNS records at a per-tenant level, not at a machine level. Companies that manage large-scale networks deal with large, broad DNS data sets. Some companies purchase supplemental DNS data sets, which can help them understand network flow patterns and security threats.

DNS logs allow for tracing suspicious activities and attack origins, so this data can be highly valuable for improving security. But DNS logs can also result in extremely large volumes of data, leading many organizations to limit their retention and compromise their security posture.

Keeping long-term DNS logs can help with:

Threat hunting: Track known malicious URLs or emails from known phishing domains, malicious command-and-control domains, or typosquatting domains.
Correlating IP visits: Identify which machines in a network accessed malicious IPs to help assess the scope of a breach and identify machines and sub-networks that may be impacted.
Detecting anomalous DNS resolutions: Some threats manipulate IP responses over time along with information on the origin of requests in order to avoid detection. These more sophisticated approaches can avoid standard detection algorithms but longer-term analysis can help spot them.
Monitoring DNS cache misses: Patterns of cache misses can indicate attempts to poison DNS records.

More and more companies will be using machine learning to automatically parse and analyze DNS query logs to aid threat hunting. With advanced persistent threats (APTs) and other sophisticated attacks, these threat-hunting tools will take a longer-term view of the data to find subtle patterns.

For example, Splunk described a DNS exfiltration detection protocol based on analyzing longer-term data, which can be used to detect more sophisticated exploits. They wrote, “By considering the context/history of communication between a host and domain and creating features to capture recent activity, we were able to identify DNS data exfiltration requests which are, by nature, slow and camouflaged between benign DNS requests.”

Using Netflow Logs to Detect Cyber Threats

Network flow data, aka “netflow,” provides metadata on network traffic as it passes through devices (such as routers, switches, and firewalls). It captures insights into who communicated with who, when those connections happened, what applications were used, and how much data was transferred. Netflow data doesn’t let you see packet and message-level information (like CDN logs do), but it does give you key information about internet traffic, which can help piece together how malicious activities and attacks unfold.

Cisco originally developed “NetFlow” in routers back in the 1990s. Since then, many other devices and systems have implemented flow data. It’s important to note that the term “netflow” is now distinct from Cisco NetFlow, and many other terms are now used to describe flow data depending on the system and provider. For example, Microsoft Azure has NSG (network security group) flow logs while AWS provides VPC (virtual private cloud) flow logs.

Here are some ways that netflow logs can help with detecting threats.

Detect long-term data exfiltration: Netflow data allows for the identification of suspicious traffic patterns, unusual activity (like large data transfers), and potential breaches.
Establish baselines: Netflow data allows for understanding normal, baseline traffic patterns for enterprises, which in turn allows for customized algorithms to detect deviations.
Combine with other key data: Combined with DNS query logs, netflow data can give you better visibility into how suspicious and potentially malicious IP addresses are interacting with your networks.
Look for patterns at user level: Netflow data can be used to create detailed user and user-group profiles. These baselines can then be used to help speed up detection of anomalous, suspicious deviations.
Detect internal policy violations. Analyzing organization-level netflow patterns can help identify violations of internal policies and rules, which can be especially important for some sensitive industries.

Just as with DNS logs, we’ll continue to see creative ideas for making use of large, historical netflow datasets to aid in detecting new and sophisticated attack patterns. For example, a 2022 paper described using machine learning applied to a very large netflow dataset to more accurately detect intrusions into internet-of-things (IoT) networks. And this 2022 paper described an idea for using netflow data to detect DNS hijacking based on detecting suspicious uses of lesser-known public DNS resolvers.

CDN Logs to Monitor for Cyber Threats

CDN logs provide information on data requests and content delivery, making them crucial for monitoring traffic patterns and detecting misuse. These logs are especially relevant for organizations that deal with massive data volumes, often using multiple CDNs, like multimedia streaming platforms.

Some security benefits of longer-term CDN log retention and analysis include:

Identifying malicious traffic surges: Abnormal patterns can include high traffic from specific IP addresses, traffic containing many invalid or spoofed packets, and surges on specific ports or protocols.
Spotting unusual download patterns: When users simultaneously download all available bitrates of a media file, that may indicate piracy or a DDoS attack.
Correlating cross-CDN attacks: Some large organizations use multiple CDNs, which can make it hard to piece together more subtle attacks. Combining logs across all CDNs can help identify coordinated attacks.
Compliance: CDN logs usually include metadata about internal security protocols and pen-testing. CDN logs can help document security best practices that are being followed, which can be relevant for industry compliance and insurance purposes.

Many organizations that use CDNs use built-in features of CDN providers to help prevent issues like DDoS (distributed denial of service) attacks. However, organizations often aren’t using historical CDN log data to gain insights into vulnerabilities and attack patterns, or to try to proactively detect data breaches and other exploits—a missed opportunity that can lead to significant security gaps. For more on this topic, read these best practices for monitoring CDN security.

WAF Logs to Detect Malicious Activity

Web Application Firewalls (WAF) monitor incoming HTTP/HTTPS requests, and WAF rules can be applied to detect and block common attack vectors. WAFs operate at the level of HTTP requests, distinguishing them from network firewalls, which filter network traffic.

While WAFs are widely used, they do have some limitations. Some WAF setups can inadvertently weaken security by providing more surface area and vulnerabilities to attacks. However, web application firewalls are still often an important part of an organization’s comprehensive application and network security posture.

WAFs are configured with rules about how each WAF should handle incoming requests and what requests should be blocked. Some WAFs come with built-in rules; WAFs like AWS WAF include third-party vendors that can implement additional pre-configured rules for your application. It’s also typical to include rules customized for the patterns in specific industries or organizations.

As with DNS logs and netflow logs, WAF logs can be high volume, leading to many organizations discarding that data after a short period of time. But WAF logs can be a valuable source of security insights. Longer-term WAF log storage can help you:

Track malicious activity over time: WAF logs provide a record of all HTTP requests, including malicious attempts. Long-term analysis helps detect “slow and low” exploits.
Fine-tuning WAF configuration: WAFs can easily be configured suboptimally; bad configuration can even be a vulnerability itself. Using historical and recent log data can help optimally fine-tune a WAF.
Cross-reference with other logs: Enhance threat detection by combining WAF data with DNS and netflow insights.

And in the short term, monitoring WAF logs can be especially helpful for updating WAF rules on the fly in response to novel attacks, DDoS attacks, and more.

For more on this topic, read about the benefits of monitoring WAF logs.

Advanced Network Log Analysis Strategies and Use Cases

Threat hunting involves searching through networks for abnormal behavior that can indicate potential attacks or even provide breadcrumbs for attacks that have already occurred. In order to proactively threat-hunt logs, long-term data retention is required. These logs can also be fed into security systems and machine learning (ML) models to improve anomaly detection and continually improve an organization’s ability to threat hunt using machine learning.

Advances in ML will make threat-hunting technology an increasingly standard resource used to parse large datasets, find abnormal patterns, and make security adjustments. Here are some ways that ML can aid in proactive threat hunting:

Using data to train security systems: The more data to work with and learn from, the better ML models can predict and prevent future attacks.
Pattern recognition across time: Detecting subtle correlations between past security incidents and present anomalies helps organizations proactively strengthen defenses.
Proactive threat hunting: Especially for larger companies, security personnel will review current known threats and vulnerabilities, and use ML to aid in the search of indicators of malicious activity and compromise.

Organizations retaining large historical datasets for analysis need solutions that provide long-term, cost-effective storage combined with “hot” data analysis that provides low-latency results regardless of the age of the data being queried.

Store and Query Long-Term Network Logs

Traditionally, it has been cost-prohibitive to store and query high-volume logs long-term, and many observability and cybersecurity platforms struggle when it comes to ingesting and analyzing the volume of data that modern enterprise companies can generate. Hydrolix is designed for real-time analytics at petabyte scale. Because Hydrolix uses decoupled, cost-effective object storage along with high-density compression that typically achieves 10x-50x compression rates, enterprises can store data long-term for a fraction of the cost of other platforms. And Hydrolix maximizes the performance of object storage, providing subsecond query analytics regardless of whether data is a minute or a year old.

Many enterprises are already using Hydrolix for monitoring. For example, Hydrolix helped FOX Sports monitor the biggest football game of the year, which had a peak ingestion rate of more than 17GB per second.

With Hydrolix, enterprises can get better security insights and operational efficiency without the high costs typically associated with long-term data retention.