A security data lake is a centralized, scalable repository that ingests, stores, and makes queryable large volumes of security-relevant telemetry — logs, network flows, endpoint events, cloud audit trails, and threat intelligence — in open formats at object-storage scale. Unlike tightly coupled SIEM platforms, a security data lake decouples raw data storage from the analytics and detection layers that sit above it, letting engineering teams retain high-fidelity, unfiltered data for months or years while applying compute selectively when they need it.

Organizations care because security data volumes are growing faster than security budgets. Traditional SIEM platforms charge per gigabyte ingested or per event per second, so teams routinely drop low-priority log sources such as CDN traffic, DNS query logs, and VPC flow records to control costs. That filtering creates blind spots that attackers exploit. A security data lake flips the economics: store everything cheaply in columnar, open formats like Apache Parquet, query selectively, and run both automated detections and ad-hoc threat hunting against the same full-fidelity dataset.

Security teams face a structural tension: regulators want more data retained for longer, attackers are harder to detect with incomplete telemetry, and budgets are finite. PCI-DSS v4.0 requires audit log retention for at least 12 months, with three months immediately available for analysis. NIST SP 800-92 recommends planning log storage in years, not months. Meeting those requirements in a hot-tier SIEM is expensive; meeting them in a security data lake built on object storage is not.

The cost argument accelerates at scale. The global SIEM market was valued at approximately $4.5 billion in 2022 and is projected to exceed $8 billion by 2027, yet much of that spend goes toward storing data that teams rarely query in real time. A security data lake lets engineering teams route high-volume, lower-priority sources such as CDN access logs, DNS, and VPC flows to cheap object storage while keeping high-priority alerts in the SIEM. This SIEM-tiering pattern reduces per-volume ingestion costs without sacrificing detection coverage.

Beyond cost, the architecture enables use cases that SIEM platforms handle poorly. Training anomaly-detection or user-behavior analytics models requires full-fidelity historical data, not a sampled or filtered subset. Retrospective investigation of a supply-chain compromise might require reconstructing attacker timelines across six months of raw endpoint, network, and cloud audit data simultaneously. A security data lake stores all of it in queryable form; a SIEM typically does not.

Two standards define the modern security data lake stack. The Open Cybersecurity Schema Framework (OCSF), backed by AWS, Splunk, IBM, CrowdStrike, and more than a dozen other founding organizations, provides a common event taxonomy for authentication events, network activity, endpoint behavior, and more. Normalizing heterogeneous log sources into OCSF at ingestion time means every downstream query, detection rule, and ML model speaks the same schema regardless of source.

Apache Iceberg provides the table format layer. Its support for ACID transactions, hidden partitioning, schema evolution, and time-travel queries makes it well-suited to security datasets that get enriched or corrected after initial ingestion. Amazon Security Lake uses exactly this combination: OCSF normalization, Parquet encoding, Iceberg table format, and S3 as the storage layer.

Detection-as-code workflows translate naturally to a security data lake. Sigma rules are open, vendor-agnostic YAML detection patterns that tools like pySigma can transpile to SQL, KQL, or SPL. A team stores normalized OCSF/Parquet data in the lake, converts their Sigma rule library to SQL, schedules those queries as batch or micro-batch jobs, and generates alerts without paying SIEM ingestion costs for every underlying log line. The MITRE ATT&CK framework v14 catalogs 196 techniques and 411 sub-techniques; mapping detection rules to ATT&CK coverage gaps is far more tractable when the underlying data is complete rather than filtered.

Engineering teams encounter three recurring friction points when building a security data lake.

Schema normalization debt. Dozens of log sources ship different field names for the same concept (source IP, user identity, timestamp). Without a normalization layer at ingestion — ideally OCSF or an internal canonical schema — queries become brittle and detection rules require per-source tuning. Investing in normalization upfront pays compound interest over time.

Small-file proliferation. High-frequency log sources writing many small Parquet files to object storage degrade query performance and inflate metadata overhead. Iceberg compaction jobs and smart time-based partitioning address this, but teams that skip compaction discover the problem only after query latency spikes.

The __LINK0__ effect. As per-GB storage and query costs fall, organizations retain more data, and total spend can rise unexpectedly. Governance policies — time-to-live rules, hot/warm/cold tiering, field-level filtering for redundant verbose sources — prevent the lake from becoming an unmanaged cost center.

A practical starting point is a narrow vertical slice rather than a full platform migration. Pick one high-volume log source that you currently drop or sample in your SIEM — VPC flow logs or CDN access logs are common candidates — and build the ingestion pipeline, schema, and a single detection query end to end.

For the ingestion layer, OpenTelemetry Collector provides vendor-neutral log collection with exporters to object storage, and the OTLP-based pipeline pattern works well for routing security telemetry without vendor lock-in. Fluent Bit is a lightweight alternative for forwarding from hosts and containers. Apache Kafka with an S3 sink connector handles high-throughput streaming scenarios.

Store data in Parquet, partition by time and source type, and apply an Iceberg table format from day one so you can add schema evolution and compaction later without a rewrite. Then connect a query engine — Trino, DuckDB, Spark SQL, or a managed equivalent — and run your first Sigma-derived SQL detection. Validate that the results match what your SIEM produces, then expand coverage incrementally. The log analytics architecture patterns that apply to observability workloads transfer directly to this use case.