HYDROLIX BLOG

Ponderings, insights and industry updates

Recent Blog Posts

Hydrolix Streaming Intake for Fastly CDN Logs

February 17, 2021

Author: Tobin Sears | Sales Engineering, Hydrolix

Tags:

In this post, we’ll show you how to configure Hydrolix to ingest your streaming Fastly CDN logs.

This post is part of a series showing how to use Hydrolix and an open source dashboard to maximize your Fastly CDN observability quickly, cheaply, and in your own VPC. Before following these quick instructions for configuring Hydrolix, check out how to configure Fastly to stream logs into Hydrolix with an HTTPS endpoint. After you have configured both Fastly and Hydrolix, check out how to analyze the transaction logs and build a dashboard to complete your observability needs.

Setup Hydrolix Streaming Intake

Please note that the following configuration assumes that you have already created a project and an associated table. If you haven’t completed those steps yet, you can follow the steps found in the Building Your Data Store section of the Hydrolix documentation website.

Create and Apply a Transform

Once you have a project and table created, the last remaining step to getting Fastly data into the Hydrolix platform is to create a transform. Think of a transform as a schema that helps tell the system what type of data to expect and ultimately how to deal with it. You can read all about the flexible Hydrolix transforms in our detailed developer docs. For this example, though, we created a transform below that you can cut and paste and use once you change the table, name, and description fields to your values. Publish the transform as you normally do with Hydrolix (or see how to publish a transform in our developer docs).

Fastly Transform (click to expand)

Leveraging Views

Hydrolix is now configured to accept the incoming Fastly log data.

Hydrolix supports the notion that a single data set can have many different query formats. The query data structure, or view, associated with a given data set not only allows for a customized representation of the queried data but also for a user’s access to the data to be restricted to a set of columns. Upon transform creation, Hydrolix automatically generates a default view that can be used to immediately query the data set – no additional configuration is required. However, users are encouraged to spend some time becoming familiar with the view concept and subsequent benefits that the feature can provide. You can refer to the developer docs for more information about query views.

Querying and Visualizing Fastly Logs with Hydrolix

Now that both Hydrolix and Fastly are configured to stream Fastly real-time log data into Hydrolix, it’s time to make some queries! Read on to check out how to analyze the transaction logs and build a dashboard to complete your observability needs.

Share Now