RSS

Why Your Log Data Should Be in a Virtual Private Cloud (VPC)

Learn about the benefits of storing your log data in a VPC, including cost savings, long-term data retention, security, and compliance.

Franz Knupfer

Published:

Dec 12, 2023

8 minute read

Is your log data truly secure? If you’re working in an industry where high levels of security, privacy, and compliance are required, the answer can’t be probably or almost definitely. It needs to be yes. That means you need fine-grained control over access to your network, including firewalls, identity and access management (IAM), and subnets. You also need full visibility into your system and the audit trails to prove you are in compliance. 

If you’re using an external observability platform to store your log data, you don’t have that level of control or visibility, so you can’t ensure your log data is truly secure. Industries ranging from healthcare to security shouldn’t settle for this level of uncertainty.

Fortunately, there is another approach. Solutions like Hydrolix allow you to ingest, store, and query your log data in your own virtual private cloud (VPC). With a VPC, you get full visibility and control when it comes to security. Because Hydrolix runs entirely on your cloud infrastructure, it is zero-egress by design. By default, data never leaves your system, making it more secure.

In this post, you’ll learn about:

  • The advantages and disadvantages of using a VPC to store your log data
  • How using a VPC with Hydrolix can ensure your data is secure—and reduce your costs

The Advantages of Using a VPC to Store Your Log Data

Using a VPC for your log data has numerous benefits—and these benefits go beyond security, compliance, and visibility. By using your own cloud infrastructure, you can dramatically reduce the cost of storing your data. Let’s take a look at each of these benefits.

Security

With a VPC, you get granular control over the security of your network, including ingress and egress. VPCs are also highly secure when set up correctly. You can set up firewalls, bastion hosts, and Network Access Control Lists (NACL) to limit your attack surface. You can enforce multi-factor authentication (MFA) and set IAM. If you need to set up communication between multiple VPCs, you can use VPC peering. And because Hydrolix runs entirely on your cloud infrastructure within a VPC, it is zero-egress by design.

In contrast, a SaaS observability platform will handle the security of your data for you. Your security and compliance teams will carefully review all potential vendors for security and compliance. And while most observability platforms make significant investments in security, you simply won’t have visibility into their system. If there is an attack or breach, you will likely learn about it in a blog or email after it happens, and the messaging may be long on PR and short on details. If you are lucky, you might get additional details of how the attack was executed and thwarted. Or you might be unlucky and your observability provider might not be aware of a breach until after it impacts your business. Regardless, you will not have the visibility and tools to monitor and avert the attack yourself.

Compliance

Some kinds of data need additional layers of compliance. An example is identifiable health information (PHI) in the United States, which needs to be HIPAA-compliant. Many observability platforms do not provide HIPAA compliance, and to further confuse matters, some platforms have products that are compliant and others that aren’t. This has the potential to create legal issues and vulnerabilities in your organization.

Observability platforms do not have a one-size-fits-all approach for security and compliance, and this is especially challenging if your business has many layers of requirements. Industries including healthcare, security, and finance need to take additional precautions to ensure that data remains both secure and in compliance. By using a VPC, you can fine-tune your configuration to ensure that it is compliant. And because Hydrolix is zero-egress by design and data never leaves your VPC, you can ensure your data is secured for even the strictest compliance standards. Note that while Hydrolix itself is SOC 2 certified and GDPR compliant, you will need to manage most compliance frameworks yourself with a VPC. For instance, a VPC is not by default HIPAA compliant, nor does Hydrolix offer HIPAA compliance—the advantage is that you have full control to ensure your VPC is in compliance by making the necessary steps within your organization.

Deploy in VPC for Increased Visibility

The only way you can have full visibility into a system is when it’s your own. With a VPC, you can log and alert on all events. You can retain logs long-term for security and compliance. You have everything you need to audit your logs. If there is a breach or attack, you have all the logs you need to investigate yourself and ensure it doesn’t happen again. With an external observability platform, you don’t have visibility into the system.

Deploy in VPC for Cost Savings

Observability costs are often a huge line item in the budget—and a major headache for decision makers at all levels. Even a fairly small cluster of machines can now generate terabytes of log data. When you add microservices, distributed systems, CDNs, and cloud infrastructure into the mix, you can easily generate logs at terabyte scale every day. Major observability platforms such as Splunk, New Relic, and Datadog offer many benefits such as all-in-one observability, but you cannot ingest terabytes of log data with these platforms at an affordable cost.

When you use a VPC to store your data, you have access to a full suite of cloud provider tools that can reduce your costs, including cost-effective object storage and the ability to more precisely scale resources to avoid overprovisioning. You aren’t paying for another provider’s infrastructure—nor are you paying for the technical debt and inefficiencies they’ve accrued along the way.

If you run Hydrolix in your own VPC, you benefit from Hydrolix’s high-density compression, which reduces your data footprint by 90%. Total cost of ownership for Hydrolix is typically 75% less than what customers previously paid to ingest, store, and query their logs. 

Hydrolix also uses Kubernetes resources within your VPC. Because Hydrolix has a stateless architecture where ingest and query are decoupled and independently scalable, you can fine-tune your compute resources to save money.

Long-term Data Retention

Long-term retention is challenging or even impossible with many observability platforms due to high costs, limitations in the platform’s storage solution, or a combination of both. You might only be able to retain data for a few weeks or even just days. This simply isn’t enough for historical analysis for many use cases, including security and compliance. For instance, a telecommunications company may be required by law to keep records of all mobile data for an extended period of time. And you may need to audit security logs months or even years later.

Logging and storing this data in a managed solution is simply too expensive—and not even possible for most observability and log management solutions, which rarely give you an option to store your log data long-term.

When you use a VPC, there are no limits on retention and you have full control over how long you store your data. The only limiter is cost, and you can use inexpensive object storage to reduce costs and maximize retention. Hydrolix also uses high-density compression to reduce your data footprint, and you also get efficient hot storage query performance whether your data is days or years old.

Flexibility

With VPCs, you get additional benefits that cloud providers offer such as horizontal and vertical scalability, the ability to set up cloud infrastructure in different regions, and a full suite of cloud-based products. This flexibility allows you to add compute for peak events and reduce compute for off-peak. You can decide exactly which regions should have resources and how to allocate those resources. With a SaaS observability platform, you don’t have as much control and flexibility to provision resources.

Pros and cons of using a VPC for log storage

Using a VPC is often the right solution if you need a higher level of  control over the security and visibility of your data. However, you should be aware of both the pros and cons of using a VPC instead of an observability platform.

ProsCons
You get granular control over security, including ingress and egress.Additional setup is required for a VPC.
You can ensure your data is in compliance with all regulatory requirements no matter how nuanced and stringent your compliance requirements are.You need to manage security yourself.
You have full visibility into all events that occur in your VPC for security and auditing purposes.If your VPC isn’t set up correctly, it may not be fully secure.
Total cost of ownership of Hydrolix with a VPC is typically 75% or more lower than other observability platforms.A VPC is just one part of a larger log management solution. You don’t get all-in-one observability like you would with an observability platform.
You have tremendous flexibility regarding the resources you add to your VPC. For instance, you can use bucket storage, which is significantly cheaper than managed storage solutions.You need additional cloud provider expertise over a SaaS solution.

Using a VPC With Hydrolix for Your Log Data

Hydrolix is a cloud data platform that runs entirely within your VPC, meaning that it’s zero-egress by design. You only need two pieces of infrastructure to use Hydrolix in your VPC:

  • Kubernetes: Hydrolix performs data ingest and query in containers in your VPC. These components are decoupled and independently scalable, giving you fine-grained control over your compute resources. Hydrolix runs on Kubernetes. You can use the Kubernetes engine your cloud provider offers, including Amazon EKS, Google Kubernetes Engine, or another Kubernetes engine.
  • Storage bucket: Hydrolix stores your data in buckets that provide cost-effective object storage. AWS, GCP, and Azure all provide data encryption for storage at rest by default, giving you an extra layer of security. Hydrolix also uses patented high-density compression to reduce your data footprint by 90%, reducing costs further.

Hydrolix comes with Grafana visualizations by default. You can also use other data visualization tools like Redash, Looker, and Superset. Your web instance is deployed inside your cloud infrastructure. Underneath the hood, the web UI directly makes API calls to your Hydrolix cluster—nothing leaves your VPC.

In addition to the security your VPC provides and Hydrolix’s zero-egress design, Hydrolix:

  • Enforces user authentication on every query and streaming endpoint.
  • Uses TLS 1.3 for secure, encrypted data.

Hydrolix is also:

  • GDPR compliant
  • SOC 2 certified

With Hydrolix, you get full control over the security and visibility of your log data. And by keeping all of your logging infrastructure inside your VPC, you can ensure that your data is truly secure.

Next Steps

Learn more about Hydrolix and contact us for a POC.

Share this post…

Ready to Start?

Cut data retention costs by 75%

Give Hydrolix a try or get in touch with us to learn more

Is your log data truly secure? If you’re working in an industry where high levels of security, privacy, and compliance are required, the answer can’t be probably or almost definitely. It needs to be yes. That means you need fine-grained control over access to your network, including firewalls, identity and access management (IAM), and subnets. You also need full visibility into your system and the audit trails to prove you are in compliance. 

If you’re using an external observability platform to store your log data, you don’t have that level of control or visibility, so you can’t ensure your log data is truly secure. Industries ranging from healthcare to security shouldn’t settle for this level of uncertainty.

Fortunately, there is another approach. Solutions like Hydrolix allow you to ingest, store, and query your log data in your own virtual private cloud (VPC). With a VPC, you get full visibility and control when it comes to security. Because Hydrolix runs entirely on your cloud infrastructure, it is zero-egress by design. By default, data never leaves your system, making it more secure.

In this post, you’ll learn about:

  • The advantages and disadvantages of using a VPC to store your log data
  • How using a VPC with Hydrolix can ensure your data is secure—and reduce your costs

The Advantages of Using a VPC to Store Your Log Data

Using a VPC for your log data has numerous benefits—and these benefits go beyond security, compliance, and visibility. By using your own cloud infrastructure, you can dramatically reduce the cost of storing your data. Let’s take a look at each of these benefits.

Security

With a VPC, you get granular control over the security of your network, including ingress and egress. VPCs are also highly secure when set up correctly. You can set up firewalls, bastion hosts, and Network Access Control Lists (NACL) to limit your attack surface. You can enforce multi-factor authentication (MFA) and set IAM. If you need to set up communication between multiple VPCs, you can use VPC peering. And because Hydrolix runs entirely on your cloud infrastructure within a VPC, it is zero-egress by design.

In contrast, a SaaS observability platform will handle the security of your data for you. Your security and compliance teams will carefully review all potential vendors for security and compliance. And while most observability platforms make significant investments in security, you simply won’t have visibility into their system. If there is an attack or breach, you will likely learn about it in a blog or email after it happens, and the messaging may be long on PR and short on details. If you are lucky, you might get additional details of how the attack was executed and thwarted. Or you might be unlucky and your observability provider might not be aware of a breach until after it impacts your business. Regardless, you will not have the visibility and tools to monitor and avert the attack yourself.

Compliance

Some kinds of data need additional layers of compliance. An example is identifiable health information (PHI) in the United States, which needs to be HIPAA-compliant. Many observability platforms do not provide HIPAA compliance, and to further confuse matters, some platforms have products that are compliant and others that aren’t. This has the potential to create legal issues and vulnerabilities in your organization.

Observability platforms do not have a one-size-fits-all approach for security and compliance, and this is especially challenging if your business has many layers of requirements. Industries including healthcare, security, and finance need to take additional precautions to ensure that data remains both secure and in compliance. By using a VPC, you can fine-tune your configuration to ensure that it is compliant. And because Hydrolix is zero-egress by design and data never leaves your VPC, you can ensure your data is secured for even the strictest compliance standards. Note that while Hydrolix itself is SOC 2 certified and GDPR compliant, you will need to manage most compliance frameworks yourself with a VPC. For instance, a VPC is not by default HIPAA compliant, nor does Hydrolix offer HIPAA compliance—the advantage is that you have full control to ensure your VPC is in compliance by making the necessary steps within your organization.

Deploy in VPC for Increased Visibility

The only way you can have full visibility into a system is when it’s your own. With a VPC, you can log and alert on all events. You can retain logs long-term for security and compliance. You have everything you need to audit your logs. If there is a breach or attack, you have all the logs you need to investigate yourself and ensure it doesn’t happen again. With an external observability platform, you don’t have visibility into the system.

Deploy in VPC for Cost Savings

Observability costs are often a huge line item in the budget—and a major headache for decision makers at all levels. Even a fairly small cluster of machines can now generate terabytes of log data. When you add microservices, distributed systems, CDNs, and cloud infrastructure into the mix, you can easily generate logs at terabyte scale every day. Major observability platforms such as Splunk, New Relic, and Datadog offer many benefits such as all-in-one observability, but you cannot ingest terabytes of log data with these platforms at an affordable cost.

When you use a VPC to store your data, you have access to a full suite of cloud provider tools that can reduce your costs, including cost-effective object storage and the ability to more precisely scale resources to avoid overprovisioning. You aren’t paying for another provider’s infrastructure—nor are you paying for the technical debt and inefficiencies they’ve accrued along the way.

If you run Hydrolix in your own VPC, you benefit from Hydrolix’s high-density compression, which reduces your data footprint by 90%. Total cost of ownership for Hydrolix is typically 75% less than what customers previously paid to ingest, store, and query their logs. 

Hydrolix also uses Kubernetes resources within your VPC. Because Hydrolix has a stateless architecture where ingest and query are decoupled and independently scalable, you can fine-tune your compute resources to save money.

Long-term Data Retention

Long-term retention is challenging or even impossible with many observability platforms due to high costs, limitations in the platform’s storage solution, or a combination of both. You might only be able to retain data for a few weeks or even just days. This simply isn’t enough for historical analysis for many use cases, including security and compliance. For instance, a telecommunications company may be required by law to keep records of all mobile data for an extended period of time. And you may need to audit security logs months or even years later.

Logging and storing this data in a managed solution is simply too expensive—and not even possible for most observability and log management solutions, which rarely give you an option to store your log data long-term.

When you use a VPC, there are no limits on retention and you have full control over how long you store your data. The only limiter is cost, and you can use inexpensive object storage to reduce costs and maximize retention. Hydrolix also uses high-density compression to reduce your data footprint, and you also get efficient hot storage query performance whether your data is days or years old.

Flexibility

With VPCs, you get additional benefits that cloud providers offer such as horizontal and vertical scalability, the ability to set up cloud infrastructure in different regions, and a full suite of cloud-based products. This flexibility allows you to add compute for peak events and reduce compute for off-peak. You can decide exactly which regions should have resources and how to allocate those resources. With a SaaS observability platform, you don’t have as much control and flexibility to provision resources.

Pros and cons of using a VPC for log storage

Using a VPC is often the right solution if you need a higher level of  control over the security and visibility of your data. However, you should be aware of both the pros and cons of using a VPC instead of an observability platform.

ProsCons
You get granular control over security, including ingress and egress.Additional setup is required for a VPC.
You can ensure your data is in compliance with all regulatory requirements no matter how nuanced and stringent your compliance requirements are.You need to manage security yourself.
You have full visibility into all events that occur in your VPC for security and auditing purposes.If your VPC isn’t set up correctly, it may not be fully secure.
Total cost of ownership of Hydrolix with a VPC is typically 75% or more lower than other observability platforms.A VPC is just one part of a larger log management solution. You don’t get all-in-one observability like you would with an observability platform.
You have tremendous flexibility regarding the resources you add to your VPC. For instance, you can use bucket storage, which is significantly cheaper than managed storage solutions.You need additional cloud provider expertise over a SaaS solution.

Using a VPC With Hydrolix for Your Log Data

Hydrolix is a cloud data platform that runs entirely within your VPC, meaning that it’s zero-egress by design. You only need two pieces of infrastructure to use Hydrolix in your VPC:

  • Kubernetes: Hydrolix performs data ingest and query in containers in your VPC. These components are decoupled and independently scalable, giving you fine-grained control over your compute resources. Hydrolix runs on Kubernetes. You can use the Kubernetes engine your cloud provider offers, including Amazon EKS, Google Kubernetes Engine, or another Kubernetes engine.
  • Storage bucket: Hydrolix stores your data in buckets that provide cost-effective object storage. AWS, GCP, and Azure all provide data encryption for storage at rest by default, giving you an extra layer of security. Hydrolix also uses patented high-density compression to reduce your data footprint by 90%, reducing costs further.

Hydrolix comes with Grafana visualizations by default. You can also use other data visualization tools like Redash, Looker, and Superset. Your web instance is deployed inside your cloud infrastructure. Underneath the hood, the web UI directly makes API calls to your Hydrolix cluster—nothing leaves your VPC.

In addition to the security your VPC provides and Hydrolix’s zero-egress design, Hydrolix:

  • Enforces user authentication on every query and streaming endpoint.
  • Uses TLS 1.3 for secure, encrypted data.

Hydrolix is also:

  • GDPR compliant
  • SOC 2 certified

With Hydrolix, you get full control over the security and visibility of your log data. And by keeping all of your logging infrastructure inside your VPC, you can ensure that your data is truly secure.

Next Steps

Learn more about Hydrolix and contact us for a POC.