Bitpacking is a compression technique that stores integers using only the minimum number of bits required to represent the range of values in a block, instead of always using the fixed width of the declared data type. If a column of 32-bit integers only contains values between 0 and 31, each value can be packed into 5 bits, shrinking the storage footprint by more than 6x before any general-purpose compressor runs on top. The idea is simple, but it sits at the core of how modern columnar engines keep query latency low while storing trillions of rows on cheap object storage.
The technique is most often applied to fixed-width integer columns inside a column block or page. The encoder scans the block, finds the maximum value, derives the bit width that covers it, and writes each value’s least-significant bits back to back into a tightly packed stream. The decoder reverses the process, often with SIMD instructions, to materialize the original integers for query execution. Bitpacking rarely runs alone. It pairs with delta encoding, frame-of-reference, dictionary encoding, or run-length encoding so the residual integers being packed are small and predictable.
Key Takeaways
- Bitpacking stores integers using only the bits needed for the value range in a block, not the full width of the declared type.
- It is most effective on low-range or low-cardinality integer columns, especially after delta or dictionary encoding has shrunk the values.
- Columnar analytics systems combine bitpacking with other lightweight codecs to cut storage cost while keeping decompression fast enough for sub-second queries.
How Bitpacking Works
A bitpacker operates on a block of integers at a time, usually 32, 64, 128, or 256 values. For each block, it finds the smallest b such that every value fits in b bits, then concatenates the b-bit representations into a contiguous byte stream. The bit width b is stored alongside the block so the decoder knows how to unpack it. This block-level adaptivity is what makes bitpacking effective on real data, where one block might need 4 bits and the next might need 17.
Block size matters for performance. SIMD-friendly implementations pack 128 integers per block to align with SSE registers, or 256 per block for AVX2, so decoding takes a handful of vector instructions per block instead of one shift and mask per value (see the widely used Lemire bitpacking library). Apache Parquet defines a hybrid encoding that combines bitpacking with run-length encoding so that repeated values collapse into a count and a value, while the rest of the page is bitpacked at a fixed width (Parquet encoding spec).
Why Pairing with Other Codecs Matters
Bitpacking only helps when the values are small. Raw 64-bit timestamps or unique IDs see little benefit because each value already uses most of the available width. Engines solve this with a pipeline. Delta encoding turns a monotonic timestamp column into small offsets between adjacent rows. Frame-of-reference subtracts a per-block base value so the residuals fit in fewer bits. Dictionary encoding maps high-cardinality strings to small integer codes that are themselves ideal candidates for bitpacking. A general-purpose codec like LZ4 or ZSTD then often runs on top of the result for a final pass (ClickHouse engineering: database compression).
Why It Matters for Analytics
Bitpacking is one of the reasons columnar formats can hold years of log and event data on object storage without breaking a budget. Hydrolix, for example, describes how its high-density compression combines bitpacking with run-length encoding, dictionary encoding, other proprietary approaches, and a final compressor to achieve very high compression rates on log data. That ratio is what makes it economical to keep every column indexed and queryable without tiering data to a slow archive.
Smaller column blocks also mean smaller byte-range reads from object storage during query execution. When the query planner pushes predicates down to storage, only the relevant blocks are fetched and decoded. The Hydrolix sparse indexing strategy works hand in hand with this. The index identifies which blocks could contain matching rows, and bitpacked blocks decompress quickly enough that network latency dominates CPU cost. The end result is the SSD-like response time on S3 described in the HDX platform overview.
Common Hurdles
The first hurdle is choosing the right block size. Small blocks adapt quickly to changes in value range, which helps compression ratio, but they add bookkeeping because each block carries its own bit width. Larger blocks amortize that overhead but may use a wider bit width than necessary when a single outlier appears in the block. Most production engines settle on 128 or 256 values per block as a reasonable balance.
The second hurdle is handling outliers. A single very large value forces the entire block to a higher bit width, wasting space on every other value. Engines mitigate this with patched approaches like PFOR (patched frame-of-reference), which pack the common range tightly and store outliers in a side list. Designing or tuning those patches requires understanding the data distribution, which is rarely uniform in practice.
The third hurdle is integration. Bitpacking is rarely a user-facing choice. It is part of an encoding pipeline that the engine selects per column based on data type, cardinality, and sort order. Schema decisions like declaring a column as uint32 instead of string, or sorting partitions by a low-cardinality key, usually have more impact on the eventual compression than any single codec setting. The Hydrolix basic data types reference shows the integer widths available, and choosing the narrowest type that fits the domain gives the encoder less work to do.
Getting Started
If you are designing a columnar schema, the most useful thing you can do for bitpacking is to be precise about types and ranges. Use the narrowest integer type that covers the domain. Sort or partition by columns with low cardinality so that adjacent rows share values, which boosts both delta encoding and run-length encoding upstream of the bitpacker. Treat strings with bounded vocabularies, like HTTP status codes or country names, as candidates for dictionary encoding so they end up as small integers.
If you are operating a Hydrolix cluster, you do not configure bitpacking directly. It is part of the HDX storage format and runs automatically inside the transform and write pipeline. The levers that matter are the schema, the primary key, and the queries. Push predicates down with WHERE clauses on indexed columns so the engine reads the smallest possible set of compressed blocks, as described in the guidance on writing efficient queries. Inspect partition sizes after ingest to confirm the compression ratio is in the expected range, and revisit schema choices for any column that looks unusually large.
FAQs
Is bitpacking the same as general-purpose compression?
No. Bitpacking is a lightweight, structure-aware encoding for integers, while general-purpose compressors like LZ4 or ZSTD treat data as a generic byte stream. Bitpacking decodes faster and integrates well with vectorized query execution, but it usually runs as one stage in a pipeline that ends with a general-purpose codec for additional savings.
Does bitpacking work on strings or floats?
Not directly. Bitpacking targets integers, so strings and floats first need to be transformed into integer representations. Strings are commonly dictionary-encoded into small integer codes, while floating-point columns may be handled by specialized codecs or by quantization before any bitpacking step.
How much space does bitpacking actually save?
It depends entirely on the value range inside each block. A column of 32-bit integers whose values all fit in 8 bits will see a 4x reduction from bitpacking alone, before any further compression. In practice, the more useful framing is the end-to-end ratio reported by the engine, which Hydrolix reports is typically 20x to 50x on log data across the full compression pipeline.
Why is decompression speed important?
Because analytic queries scan large numbers of blocks, the cost of unpacking values can become part of the query critical path. Bitpacking is designed so each block can be decoded with a few vector instructions, which keeps CPU overhead low compared to general-purpose decompressors. That headroom is what lets columnar engines hold sub-second response times even on compressed cold data.
Do I need to configure bitpacking myself in Hydrolix?
No. Bitpacking is part of the HDX storage format and is applied automatically alongside other encodings during ingest. The most effective way to influence the outcome is through schema design: choose narrow integer types, use dictionary-friendly columns for low-cardinality strings, and order partitions so that similar values cluster together.