Whether you’re streaming the Super Bowl, shopping on Black Friday, or accessing your favorite website, Content Delivery Networks (CDN) are almost certainly working behind the scenes, providing the critical infrastructure to make that experience fast and secure.
In this post, you’ll learn about the basics of CDNs, why they’re essential infrastructure for the modern internet, and the challenges (and importance) of monitoring CDN traffic.
Why CDNs Are Critical Infrastructure
CDNs have evolved from caching layers to comprehensive platforms that are now critical infrastructure for the modern internet. They solve three fundamental challenges.
- Performance: By distributing content to edge servers close to end users, CDNs dramatically reduce latency and improve the user experience, whether you’re loading a website or streaming live video.
- Scale: CDNs handle traffic spikes that would overwhelm most origin infrastructure, whether that’s Black Friday shopping surges or live sporting events with millions of concurrent viewers.
- Security: With multi-layered protection spanning DNS, application firewalls, DDoS mitigation, and data loss prevention, CDNs serve as the first line of defense against an increasingly hostile internet.
Enterprises often use a multi-CDN strategy for performance and resiliency.
How CDNs Work: The Basics
CDNs direct requests to edge servers that are close to end users, serving cached content where possible. To understand how this works, let’s start by looking at the basics of how an internet request works, including DNS resolution.
Serving an HTTP Request
When you enter a URL into your browser, four high-level steps occur:
1. DNS resolution: Your device performs a DNS lookup to translate the domain name (like hydrolix.io) into an IP address. This involves a recursive lookup that traverses the DNS hierarchy, from top-level domains like .com or .io down to the specific subdomain.
When resolving IP addresses for websites, the DNS system typically returns one of three types of records:
- A record: IPv4 address
- AAAA record: IPv6 address
- CNAME record: A canonical name that points to another domain
Note that CNAME records are much more flexible for dynamic routing, while A and AAAA records are inflexible because they are hardcoded IP addresses.
2. TCP connection: Your browser establishes a TCP connection to the server at the returned IP address.
3. HTTP request: The browser sends the actual request (such as GET or POST) to the server.
4. Response: The server sends back the requested content, which your browser renders.
While this process works, it’s not always fast or efficient, especially when your server is thousands of miles away from your users. And when you multiply this process by potentially millions of users, it becomes much more complex.
How CDNs Improve Performance
Next, let’s take a look at how CDNs use CNAME records to dynamically route traffic to edge servers.
Using CNAME to Route Traffic to Edge Server IP
When you enter a URL into your device (such as a laptop or mobile device, your device will receive a CNAME record instead of the hardcoded IP address that comes with A and AAAA records. This CNAME points to the CDN’s network. For Akamai, you’d see something like [domain].edgekey.net.
Your browser then performs another DNS lookup on that CNAME, and the CDN’s DNS system intelligently returns an IP address for an edge server based on factors like your location, network conditions, and server availability.
Connecting to a Nearby Edge Server
Next, your browser connects to an edge server, typically located in your own internet service provider (ISP) network, just a network hop or two away from your device. This proximity is crucial for performance. The farther a request has to travel, the longer it will take both to send that request and get the response.
If the requested content is already cached on the edge server, it’s returned immediately. If not, the edge server retrieves it from the origin, but it does so using the CDN’s optimized network routing, finding the fastest path across the internet to minimize network hops.
Once retrieved, the content is cached locally so subsequent requests from users in that region get lightning-fast responses. Caching content closer to end users is a critical part of CDN performance.
Security: Protecting Content Delivery
Directing traffic to edge servers and caching content are critical for content delivery, but modern CDNs do far more than just accelerate content. They’re also critical security infrastructure, and a comprehensive CDN security stack includes multiple layers.
DNS Protection
CDNs like Akamai operate thousands of distributed DNS resolvers. When you transfer your DNS zone to the CDN, DDoS attacks against your DNS infrastructure hit the CDN’s distributed system instead of attacking a single point of failure, making your infrastructure much more resilient.
Web Application Firewall (WAF)
Before requests reach your origin servers, the CDN’s WAF inspects them for common attack vectors, including:
- SQL injection attacks
- Cross-site scripting (XSS)
- OWASP top 10 vulnerabilities
WAFs are also critical for bot detection and management. Bot traffic can be benign or malicious, but even benign bot traffic can potentially overwhelm servers in some scenarios.
Origin Shielding
CDNs provide defined IP ranges (like Akamai’s Site Shield) that you can whitelist in your firewall rules. This means your origin infrastructure only accepts traffic from known CDN servers, effectively hiding you from direct internet exposure.
Network-Level DDoS Protection
For volumetric attacks that target your infrastructure at the IP level, CDNs offer scrubbing services that identify and filter malicious traffic before it reaches your network.
Data Loss Prevention
CDNs can also inspect outbound responses, checking for leaked sensitive data like credit card numbers or personally identifiable information (PII) that shouldn’t be exposed.
Streaming Media: A Different Challenge
Website delivery is one thing, but streaming live events like the NFL or the Olympics presents unique challenges. The workflow looks different, and it’s much more involved. Instead of returning static cached content, CDNs must deliver dynamic content. Let’s take a look at the process.
1. Capture and encode: Cameras capture live video, which gets transmitted to encoders that prepare it in specific codecs, typically H.264 or AAC.
2. Streaming server processing: The encoded video reaches a streaming media server that prepares it for distribution through several transformations.
- Transcoding: Converting between codec formats.
- Transmuxing: Changing container formats (such as from MOV to MP4).
- Transrating: Adjusting bitrate for different network conditions (4K at 35 Mbps down to HD at 15 Mbps).
- Transizing: Resizing video for different screen sizes (such as televisions versus mobile devices).
3. Segmentation and manifest creation: The video is chopped into segments (typically 5-10 seconds each) with a manifest file that acts as a playlist, telling players which segments to retrieve and in what order. This is also where ad insertion happens. Ad servers simply insert their segments into the manifest.
4. CDN distribution: With Akamai as an example, the live encoder pushes the prepared stream to ingest servers at the CDN, which then distribute the content to edge servers across different ISPs. Each viewer connects to an edge server in their network, receiving cached segments with minimal latency.
The Data Challenge: Why CDN Logs Matter
CDNs generate enormous volumes of log data, and retaining this data creates significant challenges. During major streaming events, enterprises must deal with massive volumes of traffic that’s sustained over several hours. For HTTP traffic, spiky traffic can make it hard to predict log volume. For example, a marketing campaign might 10x your traffic overnight, generating logs that aren’t accounted for in your budget.
Most companies can only afford to keep CDN logs for a few days before pricing becomes astronomical. However, CDN logs contain a wealth of data that’s helpful for long-term use cases.
Using CDN Data for Historical Analytics
Here are some use cases where retaining CDN data long-term (typically a year or more) is useful or even required.
Cyber Insurance Compliance
Insurance providers increasingly require proof that enterprise security practices are sufficient. Year-over-year proof of items like penetration testing results from your CDN logs can lower premiums and ensure coverage.
Low-and-Slow Attack Detection
Unlike obvious DDoS attacks, many security threats, especially from sophisticated attackers, are spread out over time in order to avoid detection. Detecting patterns of scraping, credential stuffing, or other stealthy attacks requires the ability to query months or even years of historical data.
Year-over-Year Comparisons
How did this Black Friday sale compare to last year’s? Which marketing campaigns drove the most conversions? Without long-term data retention, these questions are impossible to answer definitively.
Capacity Optimization
In order to build accurate prediction models for infrastructure planning, teams need long-term historical data. The ability to analyze past patterns helps you optimize capacity and avoid over-provisioning, directly impacting your margins.
Piracy and Content Protection
For streaming providers, detecting patterns of content theft requires tracking bot behavior over extended periods. For example, an enterprise might identify that a bot from a competitor’s IP range has been systematically scraping content over a period of months.
Conclusion
CDNs are essential infrastructure for the modern internet, handling everything from e-commerce traffic spikes to live streaming events with millions of concurrent viewers. They provide the performance, scale, and security that enterprises need, but they also create massive volumes of log data that can lead to visibility challenges.
Due to the scale of this data, most organizations can only afford to retain CDN logs for days, not months or years. This forces difficult tradeoffs between cost and capability, leaving teams without the insights they need for security investigations, compliance requirements, capacity planning, and business intelligence.
Next Steps
Fortunately, you don’t have to choose between your data and your bottom line. With solutions like CDN Insights, you can start monitoring your CDN infrastructure in minutes. Compare how multiple CDNs are performing at any scale, segment data by region, ASN, and more, and drill down into individual logs for further troubleshooting. And you can keep full-fidelity data long-term (15 months by default) without pressure to sample or throw data away.
