The Super Bowl between Philadelphia and Kansas City last month drew a record 24 million unique streaming viewers. A lot of work happened behind the scenes to ensure that viewers had an optimal experience watching the game. To support FOX with content delivery, Hydrolix provided real-time, consolidated logging across multiple content delivery networks.
It was our biggest single event to date (even bigger than last year’s big game), with Hydrolix ingesting nearly 200 terabytes of log data during the course of the three and a half hour game. Let’s get right to the numbers, which were mahoosive.
- 17.4 GB/second peak data ingest rate—the equivalent pace of a 1.4 petabyte day
- 55 billion records—lossless, high-cardinality events, all transformed, enriched, indexed, compressed and written to object storage
- 5-10 second time to glass (from event time to analysis), consistently within our target range
- 16x raw data compression rates—a 94% reduction
- 98.4% row reduction for real-time, zero-ETL, raw-data-synchronized summary tables
- 55,000 queries over the course of the event
- 0.481 second query response time (50th percentile)
That isn’t just big data—it’s monster data, at extreme volume and extreme speed, with near real-time querying and analysis. Where many observability and logging platforms will time out or simply fail at this level of scale, Hydrolix was delivering a time to glass of 5-10 seconds, including all the mechanics of receiving, standardizing, enriching, indexing, partitioning, and then writing data to cost-effective object storage. No SSDs or tightly-coupled (and expensive) storage solutions were needed to achieve this level of performance.
For this particular event, Hydrolix was tasked with ingesting, storing, and analyzing CDN log data. We’ve written a lot about multi-CDN monitoring in the past, which is particularly challenging for many observability platforms because of the sheer scale of the data, and the complexity of merging extremely disparate records into a single, logical table.
The media provider used four separate CDN vendors to ensure smooth deliverability during the event. Two of them delivered real-time logs directly to our HTTP-based stream ingest API. One used a combination of Kinesis and AWS Lambda functions to bring data to us, and one required our micro-batch autoingest feature to read files as they were being written into an S3 bucket.
The Hydrolix data infrastructure for this event was hosted on AWS’s EKS (managed Kubernetes) platform (Hydrolix is compatible with all major cloud providers). In preparation for Sunday, we deployed two Kaiju clusters for this monster data. Kaiju is what we have affectionately called the huge, multi-armed Hydrolix clusters that can hyperscale—just like the giant beasts from Japanese Kaiju movies.
The Hydrolix platform supports workload separation, so each arm of a Kaiju cluster can be targeted to handle different workloads. This allows us to tune each workload separately to maximize performance. For this event, each Kaiju cluster had arms for core, query, and intake to handle ingesting, storing, compacting, optimizing, and analyzing data. The infrastructure also included a regular cluster for video encoder logging, but this was much smaller and handled millions, not billions, of records.
In total, the Kaiju clusters ran 2400 intake head pods with a similar number of compaction workers (merge peers). The query architecture was deployed across several pools of query resources. This ensured that no users or dashboards had issues with resource contention, also known as noisy neighbors. For example, users conducting ad-hoc investigations had access to a dedicated query pool.
Hydrolix uses intake head pods to receive data and these are responsible for applying transforms, customizations, enrichments, indexes, and partial aggregations to incoming data. These pods also write this data to small partitions in object storage so they’re ingested and available for querying as quickly as possible. Meanwhile, merge infrastructure is responsible for optimizing partitions in storage, leading to higher compression rates and better query performance.
Each dashboard queried the previous 30 minutes by default. In a 30-minute window, we gathered on average 19 billion rows of raw data. We handled that volume efficiently with summary tables, which aggregate the metrics needed for dashboards and shrink the row count by as much as 98–99.99% via repeated merges (see below). At the same time, the underlying raw data stays fully accessible for ad hoc queries, which are still efficient on Hydrolix’s optimized object storage. By cleaning our data in advance and relying on these merges, we kept dashboards fast and resource-efficient—crucial during a high-traffic event.
For query capacity, one Kaiju included 15 query heads (60 CPU/170Gi) and 85 query peers (60 CPU/170Gi) and another had 15 query heads (60cpu/120Gi) and 200 query peers (28cpu/60Gi). They ran on compute-optimized c5n.18xlarge and c6in.16xlarge instances. Since the customer primarily used summary tables, query compute was extremely efficient, amounting to 8.6 cores per terabyte of data ingested—and this led to a p50 query latency of .481 seconds over all dashboards.
And due to the fact that Hydrolix uses stateless compute, we were able to fully scale down after the event was complete, leading to additional savings and reducing overall compute usage. Meanwhile, all the raw and summarized data remains safe and sound in object storage for later review.
This game was a lot of work—and a big success—for our engineering and customer success teams. But we were also able to have some fun on the side, too. We used an AI tool to help summarize our conversations as we were monitoring our systems for around 15 hours, and it came up with a billing and capacity planning metric we never would’ve expected: RpCW, rows per chicken wing. It’s based on how many chicken wings are eaten during the game. Skynet, eat your heart out!
We’re not just able to deliver this level of performance, but we also do it cost-effectively by building on object storage. We can do log data at hyperscale—petabytes per day—without it costing the whole bird…
Next Steps
If you’re not using Hydrolix yet and would like to learn more, contact Hydrolix about a managed trial or demo.
