In this blog post, we’ll show you how to configure Fastly to stream logs to Hydrolix with just a few clicks and in less than 10 minutes.
This post is part of a series showing how to use Hydrolix and an open source dashboard to maximize your Fastly CDN observability quickly, cheaply, and in your own VPC. Start here to configure Fastly to stream log data to Hydrolix, then configure the Hydrolix streaming intake for those logs, see how to analyze Fastly transaction logs using Hydrolix, and learn how to visualize Fastly log data with an open source dashboard. You can also always refer to our Hydrolix documentation.
Hydrolix supports native integration with Fastly’s real-time log streaming service via an HTTPS Endpoint. With Hydrolix and an open source dashboard, you can maximize your Fastly CDN observability quickly, cheaply, and in your own VPC.
Similar documentation can also be found at the Fastly Integrations website: Fastly Log Streaming Guide for Hydrolix
Table of Contents
Setup Fastly Log Streaming: HTTPS
Prerequisites
When sending logs to an HTTPS endpoint, Fastly requires proof that you control the domain name specified in the endpoint URL field. This is accomplished by using an HTTP challenge on a well-known path. Fortunately, Hydrolix has a pre-configured path for quick validation:
https://{{your_instance}}.hydrolix.live/.well-known/fastly/logging/challenge
By default, Hydrolix uses an asterisk (*) to allow any service to post to the HTTP endpoint.
Configure the HTTPS Logging Endpoint
In this example, we will create a new HTTPS logging endpoint that will be used to send real-time streaming logs to Hydrolix.
1. Login to Fastly
Login to manage.fastly.com and choose the appropriate service.
2. Create an HTTPS Logging Endpoint
Under the desired Fastly service configuration, select: * Logging * HTTPS Endpoint
Enter the following configuration options:
(For reference, the 14-field default log format for a Fastly HTTPS Logging Endpoint can be found here.)
Name: Name for the Hydrolix logging endpoint
Log format: In order to better leverage the value provided by the Hydrolix platform from a data compression and query performance perspective, we will be using an extended version of the Fastly streaming log format that increases the logged fields from the default of 14 to 64 in our example below. Below is the complete extended log format that you can cut and paste for our example.
Extended Fastly Log Format (click arrow to expand)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 |
<span class="token punctuation"><br>{</span> <span class="token property">"service_id"</span><span class="token operator">:</span> <span class="token string">"%{req.service_id}V"</span><span class="token punctuation">,</span> <span class="token property">"service_version"</span><span class="token operator">:</span> <span class="token string">"%{fastly_info.version}V"</span><span class="token punctuation">,</span> <span class="token property">"time_start"</span><span class="token operator">:</span> <span class="token string">"%{begin:%Y-%m-%dT%H:%M:%S}t"</span><span class="token punctuation">,</span> <span class="token property">"time_end"</span><span class="token operator">:</span> <span class="token string">"%{end:%Y-%m-%dT%H:%M:%S}t"</span><span class="token punctuation">,</span> <span class="token property">"time_elapsed"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>time.elapsed.usec<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"client_ip"</span><span class="token operator">:</span> <span class="token string">"%{req.http.Fastly-Client-IP}V"</span><span class="token punctuation">,</span> <span class="token property">"request"</span><span class="token operator">:</span> <span class="token string">"%{req.request}V"</span><span class="token punctuation">,</span> <span class="token property">"protocol"</span><span class="token operator">:</span> <span class="token string">"%{req.proto}V"</span><span class="token punctuation">,</span> <span class="token property">"host"</span><span class="token operator">:</span> <span class="token string">"%{req.http.Fastly-Orig-Host}V"</span><span class="token punctuation">,</span> <span class="token property">"origin_host"</span><span class="token operator">:</span> <span class="token string">"%{req.http.Host}V"</span><span class="token punctuation">,</span> <span class="token property">"url"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.url)}V"</span><span class="token punctuation">,</span> <span class="token property">"is_ipv6"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>if(req.is_ipv6<span class="token punctuation">,</span> <span class="token string">"true"</span><span class="token punctuation">,</span> <span class="token string">"false"</span><span class="token punctuation">)</span><span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"is_tls"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>if(req.is_ssl<span class="token punctuation">,</span> <span class="token string">"true"</span><span class="token punctuation">,</span> <span class="token string">"false"</span><span class="token punctuation">)</span><span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"tls_client_protocol"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(tls.client.protocol)}V"</span><span class="token punctuation">,</span> <span class="token property">"tls_client_servername"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(tls.client.servername)}V"</span><span class="token punctuation">,</span> <span class="token property">"tls_client_cipher"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(tls.client.cipher)}V"</span><span class="token punctuation">,</span> <span class="token property">"tls_client_cipher_sha"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(tls.client.ciphers_sha )}V"</span><span class="token punctuation">,</span> <span class="token property">"tls_client_tlsexts_sha"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(tls.client.tlsexts_sha)}V"</span><span class="token punctuation">,</span> <span class="token property">"is_h2"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>if(fastly_info.is_h2<span class="token punctuation">,</span> <span class="token string">"true"</span><span class="token punctuation">,</span> <span class="token string">"false"</span><span class="token punctuation">)</span><span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"is_h2_push"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>if(fastly_info.h2.is_push<span class="token punctuation">,</span> <span class="token string">"true"</span><span class="token punctuation">,</span> <span class="token string">"false"</span><span class="token punctuation">)</span><span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"h2_stream_id"</span><span class="token operator">:</span> <span class="token string">"%{fastly_info.h2.stream_id}V"</span><span class="token punctuation">,</span> <span class="token property">"request_referer"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.Referer)}V"</span><span class="token punctuation">,</span> <span class="token property">"request_user_agent"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.User-Agent)}V"</span><span class="token punctuation">,</span> <span class="token property">"request_accept_content"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.Accept)}V"</span><span class="token punctuation">,</span> <span class="token property">"request_accept_language"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.Accept-Language)}V"</span><span class="token punctuation">,</span> <span class="token property">"request_accept_encoding"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.Accept-Encoding)}V"</span><span class="token punctuation">,</span> <span class="token property">"request_accept_charset"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.Accept-Charset)}V"</span><span class="token punctuation">,</span> <span class="token property">"request_connection"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.Connection)}V"</span><span class="token punctuation">,</span> <span class="token property">"request_dnt"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.DNT)}V"</span><span class="token punctuation">,</span> <span class="token property">"request_forwarded"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.Forwarded)}V"</span><span class="token punctuation">,</span> <span class="token property">"request_via"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.Via)}V"</span><span class="token punctuation">,</span> <span class="token property">"request_cache_control"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.Cache-Control)}V"</span><span class="token punctuation">,</span> <span class="token property">"request_x_requested_with"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.X-Requested-With)}V"</span><span class="token punctuation">,</span> <span class="token property">"request_x_forwarded_for"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(req.http.X-Forwarded-For)}V"</span><span class="token punctuation">,</span> <span class="token property">"status"</span><span class="token operator">:</span> <span class="token string">"%{resp.status}V"</span><span class="token punctuation">,</span> <span class="token property">"content_type"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(resp.http.Content-Type)}V"</span><span class="token punctuation">,</span> <span class="token property">"cache_status"</span><span class="token operator">:</span> <span class="token string">"%{regsub(fastly_info.state, "</span>^(HIT-(SYNTH<span class="token punctuation">)</span>|(HITPASS|HIT|MISS|PASS|ERROR|PIPE<span class="token punctuation">)</span><span class="token punctuation">)</span>.*<span class="token string">", "</span>\\<span class="token number">2</span>\\<span class="token number">3</span><span class="token string">")}V"</span><span class="token punctuation">,</span> <span class="token property">"is_cacheable"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>if(fastly_info.state ~<span class="token string">"^(HIT|MISS)$"</span><span class="token punctuation">,</span> <span class="token string">"true"</span><span class="token punctuation">,</span> <span class="token string">"false"</span><span class="token punctuation">)</span><span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"response_age"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(resp.http.Age)}V"</span><span class="token punctuation">,</span> <span class="token property">"response_cache_control"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(resp.http.Cache-Control)}V"</span><span class="token punctuation">,</span> <span class="token property">"response_expires"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(resp.http.Expires)}V"</span><span class="token punctuation">,</span> <span class="token property">"response_last_modified"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(resp.http.Last-Modified)}V"</span><span class="token punctuation">,</span> <span class="token property">"response_tsv"</span><span class="token operator">:</span> <span class="token string">"%{cstr_escape(resp.http.TSV)}V"</span><span class="token punctuation">,</span> <span class="token property">"geo_datacenter"</span><span class="token operator">:</span> <span class="token string">"%{server.datacenter}V"</span><span class="token punctuation">,</span> <span class="token property">"geo_city"</span><span class="token operator">:</span> <span class="token string">"%{geoip.city}V"</span><span class="token punctuation">,</span> <span class="token property">"geo_country_code"</span><span class="token operator">:</span> <span class="token string">"%{geoip.country_code}V"</span><span class="token punctuation">,</span> <span class="token property">"geo_continent_code"</span><span class="token operator">:</span> <span class="token string">"%{geoip.continent_code}V"</span><span class="token punctuation">,</span> <span class="token property">"geo_region"</span><span class="token operator">:</span> <span class="token string">"%{geoip.region}V"</span><span class="token punctuation">,</span> <span class="token property">"req_header_size"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>req.header_bytes_read<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"req_body_size"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>req.body_bytes_read<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"resp_header_size"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>resp.header_bytes_written<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"resp_body_size"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>resp.body_bytes_written<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"socket_cwnd"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>client.socket.cwnd<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"socket_nexthop"</span><span class="token operator">:</span> <span class="token string">"%{client.socket.nexthop}V"</span><span class="token punctuation">,</span> <span class="token property">"socket_tcpi_rcv_mss"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>client.socket.tcpi_rcv_mss<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"socket_tcpi_snd_mss"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>client.socket.tcpi_snd_mss<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"socket_tcpi_rtt"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>client.socket.tcpi_rtt<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"socket_tcpi_rttvar"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>client.socket.tcpi_rttvar<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"socket_tcpi_rcv_rtt"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>client.socket.tcpi_rcv_rtt<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"socket_tcpi_rcv_space"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>client.socket.tcpi_rcv_space<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"socket_tcpi_last_data_sent"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>client.socket.tcpi_last_data_sent<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"socket_tcpi_total_retrans"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>client.socket.tcpi_total_retrans<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"socket_tcpi_delta_retrans"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>client.socket.tcpi_delta_retrans<span class="token punctuation">}</span>V<span class="token punctuation">,</span> <span class="token property">"socket_ploss"</span><span class="token operator">:</span>%<span class="token punctuation">{</span>client.socket.ploss<span class="token punctuation">}</span>V <span class="token punctuation">}</span> |
URL: The streaming API endpoint for your Hydrolix instance: https://<hydrolix-instance-name>.hydrolix.live/ingest/event
Maximum logs: 0
Maximum bytes: 0
EXPAND “Advanced options”
Content type: application/json
Custom header name: x-hdx-table
Custom header value: <hydrolix_project_name>.<hydrolix_table_name>
Method: POST
JSON log entry format: Newline delimited
Select a log line format: Blank
Placement: Format Version Default
Using your own certificate authority (CA)?
Leave all values empty or customize as required
Congratulations! Fastly has now been configured to send real-time streaming logs to the Hydrolix endpoint. Read on to see how to configure Hydrolix to receive the Fastly logs.