Lately, we’ve been talking a lot about mahoosive data at Hydrolix…
Recently, I was asked what we mean by mahoosive data. If you haven’t heard the term “mahoosive” before, it’s a British way of saying very, very, very big. In the case of Hydrolix, that means ingesting very large amounts of log data at high velocity and high levels of cardinality, with a need to get eyes on that data as fast as possible. The events in question are some of the biggest sporting events in the world (and yes, managing CDN log data for FOX for the Super Bowl was one of those events).
These events (and they continue to grow) are generating in excess of 15 million rows of log data per second. In the case of the Super Bowl, Hydrolix managed upwards of 17.4 GB/second—with nearly 200 terabytes of data collected over the course of the event.
The term “managed” is a bit of a (typically British) understatement. There’s a lot going on under the hood when it comes to ingesting data.
Hydrolix’s streaming ingest mechanism receives a bunch of rows in a single message, decompresses those rows, extracts them from the format they were sent in, performs health heuristics to ensure the data is “good” data, and then transforms the data based on user-specified transforms.
But wait, we’re just getting started.
The ingest mechanism includes a transformation process where we edit and clean the data, which includes everything from truncating, appending, looking up, munging, and normalizing to extracting substrings based on regular expressions.
The data must then be indexed and compacted, which means sorting it, indexing all fields regardless of cardinality, separating it into partitions, and then writing those partitions (using massive parallelism) to object storage. And that data can be sent to multiple buckets, all while enabling the ingest from many different sources into a single table for hassle-free comparisons without complex JOIN statements.
But wait, there’s more. In addition to writing the underlying raw data to object storage, the data is simultaneously written to calculated summary tables that hold intermediate states for on-the-fly, accurate aggregations. These summary tables are much smaller than the underlying raw data, resulting in efficient, high-performance querying for dashboards and users that need high-level metrics.
Total time to glass from the moment a log message arrives until it can be queried: 5-10 seconds. And that’s even the case when the system is streaming 15 million log lines per second.
Just How Much Data Is That? (A Hitchhiker’s Guide)
When it comes to defining mahoosive data at this scale, it’s time to get the abacus out.
Considering that the average size for each CDN log line that Hydrolix ingests is around 1200 bytes…
At 15 million log lines per second, that amounts to approximately 18 billion bytes per second (right around what we were seeing for the Super Bowl).
Now let’s translate that from digital into paperback—or rather, let’s translate a very specific paperback book into digital. One of my favorite books is The Hitchhikers Guide to the Galaxy, which is about 210 pages (depending on the edition).
If the typical paperback book has about 32 lines of text per page and 65 characters per line, that’s about 2080 bytes per page.
The answer to the universe may be 42. Meanwhile, the answer to how many bytes per hitchhiker’s guide is approximately 436,800 bytes (2080 x 210), or about 437 kilobytes (KB). I found a version online that is 432 KB, but let’s just say in my version it has some extra fancy footnotes that talk about what a good book it is.
At 18 billion bytes per second, and with each copy about 437 KB, that would be ingesting 41,190 copies of The Hitchhiker’s Guide to the Galaxy each second. Imagine that each copy is made up of log files that come in from different sources and in different formats, and these files need to be decompressed, read, checked, cleaned, edited, enriched, sorted, indexed, partitioned, recompressed, and then written to storage—all with a 5-10 second time to glass for us human mortals to be able to properly analyze and read.
Over a two-hour event, that amounts to nearly 300 million copies of The Hitchhiker’s Guide to the Galaxy—true bestseller status! (As of 2005, there have been 14 million copies of the book sold.)
That doesn’t even account for all the additional data manipulation, calculation, and transformation required for aggregating raw data into summaries for readers and analysts that aren’t so engrossed in the guide—so throw in the equivalent of a few summaries for the book critics, a selection of Vogon Poetry for the lonely traveller, and some Cliff’s Notes of Marvin’s complaints for some light reading.
Add that all together and you get Mahoosive Data!
Now, where did I leave my towel…
Next Steps
If you’re not using Hydrolix yet and would like to learn more, contact Hydrolix about a managed trial or demo.
