Stream piracy is a huge issue for broadcasters, streaming platforms, and content providers. It’s predicted to cost streaming providers hundreds of billions of dollars in revenue in the coming years. In addition to posing major threats to revenue and intellectual property, it can reduce how much both users and advertisers trust your brand.
There are many nightmare scenarios. CDN leeching eats into revenue while stealing CDN bandwidth–all while driving CDN costs up. Pirated streams show up on malware sites, leading to ads placed next to inappropriate content. And enterprises, including CDN and streaming providers, also need to take adequate measures to prevent piracy due to the Digital Millennium Copyright Act (DMCA) or run the risk of paying big fines.
One of the most effective ways to fix stream piracy and CDN leeching is by monitoring CDN logs to detect stream piracy in real time, especially when combined with techniques like DRM (digital rights management) and watermarking. In addition to storing video and streaming assets at the edge for improved resiliency and deliverability, CDNs generate a vast amount of logs on how and when these assets are being requested and used, allowing teams to identify suspicious activity and stream piracy.
Combined with other tools such as web application firewalls (WAFs), tokenized URLs, and geo-blocking, teams can quickly and proactively shut down pirated streams and stop leeching, protecting your revenue, bandwidth, and brand.
This post will cover common CDN fields and user behaviors to monitor to quickly identify issues so you can update WAF rules, revoke access tokens, and more. It will also cover some of the challenges of monitoring CDN traffic and how Hydrolix can make it much easier to prevent piracy.
Detecting Stream Piracy and CDN Leeching in CDN Logs
The majority of enterprises use CDNs to cache content close to the edge and make media delivery more performant. This is especially important for streaming and video content, where users expect high-resolution playback without buffering and other issues. Meanwhile, large media files as well as high viewership (especially for live events) can put intense pressure on available bandwidth—and also generate a huge volume of log data. To add to these challenges, enterprises need to be able to quickly detect and block CDN leeching that siphon users, revenue, and bandwidth—all while driving up the cost of CDNs.
Fortunately, the logs CDNs generate have a wealth of information that teams can use to detect and mitigate stream piracy and CDN leeching. This information includes top IP addresses, token usage, geos, user agents, and request paths.
URL Tokens
Media playback URLs are usually tokenized, and one token is unique to one user session. If an unusually high number of people are using the same token, it indicates that the tokenized URL has been published online and many people are illegally accessing the stream.
The following Hydrolix dashboard shows tokens that have unusually high usage. In addition to seeing the total count of each token, you can easily identify the tokens, country codes, and IP addresses involved in each piracy event. This allows teams to take precise anti-piracy actions by zeroing in on the specific clients, sessions, IPs, user agents, and devices that are pirating the stream.
The following video shows how you can use Hydrolix to quickly see which tokens may be engaging in piracy, CDN leeching, and account sharing, and then one potential method of taking action by checking whether WAF rules are mismatched and need to be updated.
Top IP Addresses, Countries, and Geos
When an IP address makes a high number of requests, it’s often a sign of suspicious activity. It could be an indicator of a denial of service (DoS) attack—and malicious actors do often target major events. But it can also be evidence of stream piracy as well. Stream pirates and CDN leechers often aggressively rip content at regular intervals so they can record and reshare streams. They’ll also often download all available bitrates. This generates a high number of requests, and it’s a clear indicator of suspicious traffic (often bots). In contrast, human viewers will use media players that download a single bitrate at a time.
In Hydrolix, the following query would return the top IP addresses by count over the last hour from a table called cdn_logs.
SELECT
ip_address,
COUNT(*) AS request_count
FROM
cdn_logs
WHERE
event_time >= now() - INTERVAL 1 HOUR
GROUP BY
ip_address
ORDER BY
request_count DESC
LIMIT 10;Any abnormally high counts could indicate stream piracy or other malicious behavior such as denial of service attacks. If you’re using web application firewalls (WAF), you can then update your WAF rules to specifically omit individual IP addresses or ranges of addresses.
In addition to tracking by IP address, country and geo information can provide helpful at-a-glance information for your teams. For example, you might want to flag traffic from Belarus as suspicious. But you may also have content that’s only available to certain regions—for example, a regional provider may only provide content to a specific region, and activity outside of that region may be restricted.
If you’re seeing signs of activities from restricted regions, that’s an indicator that WAF rules need to be fine-tuned.
User Agents
Stream pirates and CDN leechers often use scraping tools and bots for ripping content. These include tools such as Scrapy or a Python script. Meanwhile, humans typically use agents associated with their devices, whether that’s browsers, mobile devices, televisions, or something else.
CDN logs typically include a user agent field. You can add this field to the SELECT statement of GROUP BY queries that aggregate top IP addresses to get additional evidence of stream piracy. Tools like scrapy, Python libraries, and headless browsers are not human users and may indicate suspicious behavior. So are requests where the user agent is blank, undefined, or an empty string. If you’re keeping WAF rules up to date, you may have already excluded requests from top scraping and bot tools already. But as new tools and user agents arise, you’ll need to modify your WAF rules to include them. Tracking user agents doesn’t just give you a sense of what malicious actors are up to. It can also help you keep your WAF rules robust.
URLs and Request Paths—Especially Combined with Honeypots
Bots and scrapers are more likely to crawl and attempt to scrape content that is restricted or expired, or they may simply attempt to visit non-existent request paths. For example, if there are request paths ranging from episode-1 to episode-10, a scraper may try to scrape episode-11 even if it doesn’t exist. You can also implement honeypots to more easily detect this kind of behavior. A very basic example might be a resource that is only visible to scrapers and bots, not to human users. Any interaction with this kind of resource is an indicator of bot traffic. A dashboard that returns regular query results of interactions with assets and URLs can be extremely useful, providing IP addresses to block or throttle.
Referrer URLs
Referrer URLs are typically stored in HTTP headers, and they provide information about the previous page that a user accessed. Media and streaming companies often have a list of legitimate referrers—for instance, the home page of the site would be a legitimate referrer, and they may have affiliates that are also legitimate referrers. Enterprises can monitor unexpected referrers, which in many cases indicate piracy. In the case of Hydrolix, the customer success team helped one major broadcaster discover that more than 50% of traffic was coming from illegitimate referrers. With this information, the broadcaster was able to take additional steps to mitigate piracy.
User Behavior
Bots and scrapers have request patterns that are very different from how a typical media player interacts with digital content. For example, when an actual viewer downloads a stream, it will typically request small chunks of a single bitrate in sequential order. In contrast, stream piracy tools may repeatedly request the same chunk or request chunks of content out of order. They may request multiple segments much faster than a human can view them. And they may request multiple or all bitrates at once.
You can also analyze unusual user behavior in on-demand content. For example, you might expect the majority of users to make asset requests during standard viewing hours such as weeknights and weekends. If you see IPs requesting an unusual number of video on demand (VOD) assets at odd hours such as 4 AM on a Monday morning, that could be a piracy event as well as an indicator that an account may have been breached. You can look for other indicators, such as past account behavior, to better understand whether the behavior is anomalous.
Sometimes, unusual patterns may only be visible over long time periods. For example, malicious users may request assets infrequently to avoid detection. Having access to long-term, high-fidelity data can help with finding and eliminating this behavior.
Using Hydrolix to Prevent Stream Piracy and CDN Leeching
Considering the high cost of piracy and leeching for many enterprises, detecting and mitigating the issue as quickly as possible needs to be a top priority—and considered on par with mission-critical use cases like application performance and brand management. However, many observability platforms are unable to handle the high volume of logs that CDNs can generate. And with traditional platforms, it can be very costly to analyze and retain such a high volume of data, which is why enterprises have traditionally discarded CDN data instead of retaining it.
To effectively detect and mitigate these issues, you need a solution designed to process high-volume CDN logs. It must be fast, scalable, and cost-effective. That’s where Hydrolix comes in.
Hydrolix has deep expertise in the CDN monitoring space, and many of the largest streaming and media providers in the world, including FOX and others, use Hydrolix for monitoring CDNs. With Hydrolix, you can standardize all your log data from multiple CDNs into a single table, making it easy to correlate data across CDNs. Time to glass is under ten seconds—even when ingesting terabytes of log data on a daily basis. You get all the high-level metrics covered in this post, plus the ability to drill down into your data with subsecond needle-in-a-haystack queries—even when querying petabytes of data.
All data in Hydrolix is “hot” for querying and stored in cost-effective object storage with fifteen months of retention by default. This can be especially helpful for detecting account sharing, which is more subtle and harder to detect than stream piracy, and for detecting “low and slow” malicious behavior that would fly under the radar otherwise. For instance, you can see which accounts have been accessed by five or more IPs in the last 90 days and mitigate account over usage. Or you can correlate your logs over a long period of time to better understand which accounts, users, and IPs are engaging in suspicious behavior. You can also use the Hydrolix Connector for Apache Spark to analyze patterns in historical CDN data for use cases like data science and AI model training.
Next Steps
- Interested in learning more about Hydrolix? Sign up for a technical demo or a free trial.
- Learn more about the value of CDN monitoring, complete with dashboards that show valuable CDN metrics.
- Learn best practices for multi-CDN monitoring.
- Learn about the value of CDN switching and why monitoring is essential for a multi-CDN strategy.
- Learn how a major streaming media provider uses Hydrolix for multi-CDN monitoring.
