AI is evolving daily. If you aren’t using, integrating, enhancing, or building AI capabilities into your services, you are falling behind. That’s why creating space for innovation is essential. Hydrolix recognized this need in 2024 when its co-founder had a laundry list of projects he wanted the team to execute. That need became the impetus for the Hydrolix Innovation Lab, led by Director Alok Aggarwal and managed by Senior Product Manager Ashley Vassell.
Today, the Innovation Lab comprises 15 engineers working on a range of projects, all designed to address customer needs. I spoke with Alok and Ashley about the team’s current projects, their rationale for selecting these projects, and the future of the Innovation Lab.
Abby: Hi, Alok and Ashley. Thank you for speaking with me today. What was the initial focus of the lab when it opened in June 2024?
Alok: Thank you, Abby. The initial focus was on bringing to market a new Splunk connector, built by Hydrolix co-founder Hasan Alayi, to integrate into Hydrolix. We’d heard from customers that they value their Splunk dashboards but struggle with the rising costs of retaining terabyte-to-petabyte-scale data. If they wanted to maintain data for more than a week, Splunk costs soared, yet they still wanted to keep their Splunk UI. We wanted to offer them the best of both worlds: continue using your Splunk dashboards, but save money by using Hydrolix on the backend.
Abby: That makes sense. Because Hydrolix can compress data 25-50x, we offer lower-cost data retention than other providers.
Alok: Yes. Due to our work in the Innovation Lab, Hydrolix has become a supplemental solution to the Splunk installation base. Additionally, recognizing that data is the foundational basis for machine learning and AI model development, we sought to provide our customers with a means to retain and query all their data affordably and quickly, regardless of size or age. That meant Hydrolix data had to be available to ML and AI engineers in their most common practice environment, which is Apache Spark. So we built a Spark connector for Databricks, Microsoft Fabric, and AWS EMR. With that connector, customers could store all their data and keep it hot. Those were the first two projects in the lab.
Watch the latest episode of Inside Hydrolix, where Hydrolix Product Marketing Manager Josh Robinson talks with Ashley Vassell about Hydrolix Innovation Lab and democratizing data with AI.
Abby: What about another challenge we often encounter with customers—the lack of readily available ML talent. How did the lab help overcome that?
Alok: We observed this trend emerge after building the connectors and decided to expand our focus. We created our own MCP server to make Hydrolix data available not only to engineers proficient in SQL but also to anyone who wants to interrogate data in natural language. That means the most non-technical people in a company, such as a CMO or HR leader, can query Hydrolix data for their own use cases.
Abby: Would you say the lab is more focused on strategic, larger solutions or tactical work?
Alok: It’s a mix of both. We selectively pick tactical work. For example, we heard from Hydrolix customers who primarily use Grafana that they faced issues with the ClickHouse Grafana plugin. To address this issue, our engineers developed a Grafana plugin for Hydrolix data sources, enhancing the query experience.
Abby: That really changes the customer experience. I’m not a SQL professional, so I appreciate that, as a marketer, I can build my own dashboard for any project within my purview. Ashley, I know you’re managing several projects in the lab today. What’s the latest one?
Ashley: As we announced a few weeks ago, Hydrolix purchased intellectual property from Quesma to continue supporting Kibana interoperability for our customers. We’re now focusing on integrating the Kibana Gateway (formerly known as the Quesma IP) into our Hydrolix data platform and adding summary tables to support it. We’re also focusing on anomaly detection. Our anomaly detection system is ML-based and focuses on correlating data from different metrics to pinpoint the root cause of an anomaly. It’s unique because, with basic detection systems, you typically only access one set of metrics, whereas Hydrolix correlates these metrics. If you see a spike in one metric and the other correlated metric is behaving differently, that indicates an anomaly. And we’re using LLMs for natural language correlation of the anomalous data.
Abby: Can you give more details? How does this differ from other anomaly detection systems at a higher level?
Ashley: With most detection systems, you receive an alert indicating a value is high, but you don’t know what it means or where to look. Hydrolix’s anomaly detection system, which we developed in the Innovation Lab, enables users to directly identify the root cause of the anomaly and take action to mitigate it. It’s action-oriented, rather than just another dashboard that people watch all the time. It’s highly accurate and solves real problems for users instead of alerting them to things they won’t care about. For example, let’s say you’re concerned about cache performance degradation. The CDN has to return to the customer’s origin more frequently to retrieve content because it’s not being cached. We detect this issue by examining how often the CDN cache is being accessed. If that number is decreasing, it means there’s an issue with the CDN provider. Hydrolix also ingests full-fidelity data, and a lot of it. We often see anomaly detection being done with sampled data, which can introduce bias or noise. Since Hydrolix takes in, retains, and analyzes all data, there’s no need to test; anomaly detection is therefore more accurate.
Abby: Going back to what Alok said about customers lacking ML talent. What are you doing with anomaly detection to solve that problem?
Ashley: Our customers don’t have to do any manual operation around our anomaly detection system. They receive alerts via email. Those alerts link to a dashboard that displays a root cause analysis report, identifying the affected region where Hydrolix detected anomalous streams and suggesting a mitigation action. Actions could be blocking IPs or improving WAF rules. No matter the issue, Hydrolix’s anomaly detection system leads customers to how to solve it.
Abby: Is this available now?
Ashley: The full solution will be completed by the end of this quarter. We’re working on it in the lab.
Abby: So what’s in store for the future of the lab?
Ashley: We’re researching agentic AI and how to bring those capabilities into Hydrolix. We’re also exploring vectorization and embeddings, which enable users to conduct more effective searches. Piracy detection is another big one. When considering bot detection, whether it’s AI scraping bots or media streaming bots stealing streams, piracy detection, and the ability to more accurately identify those instances with full-fidelity data versus using sampled data are critical.
Abby: Now that you’ve built the MCP server, what other natural language capabilities are you working on in the lab?
Ashley: Our MCP server already enables some of them. The goal is for a salesperson, for example, to interact with Hydrolix in natural language, such as “Tell me what my sales were last week.” Hydrolix would send a response with a chart. The visual would be a chatbot that’s embedded into our console. We want to add the ability to say “Make this into a dashboard,” and have it generate a dashboard, or it could stay in a chat.
Abby: I’m looking forward to trying that. Alok, what’s in store for the future of the lab itself? Will you expand it?
Alok: I’m pleased with what we’ve built and the projects we’ve delivered in such a short amount of time. From the Splunk and Spark connectors to the MCP server, anomaly detection, the Grafana plugin, and now the Kibana Gateway, we’ve released a significant number of updates. I hope that we expand these—and those in the pipeline—to more markets globally. We aim to deliver end-to-end, integrated, turnkey solutions that address a set of key pain points.
Abby: Thank you both for speaking with me today. To learn more about Hydrolix’s latest connectors, which were built in the lab, and the importance of retaining all high-dimensionality, high-cardinality data, read this post on AI and Jevons Paradox.
Next Steps
- Watch Inside Hydrolix Episode 4: Hydrolix Innovation Lab: Democratizing Data with AI.
- Learn how Hydrolix is helping to build the next generation of AI infrastructure.
- Learn about using the Hydrolix Connector for Apache Spark (Databricks), including how the connector works and how Hydrolix performs against Delta Lake.
- Read about the latest updates to the Hydrolix Connector for Apache Spark.
- Contact Hydrolix about a proof of concept or demo.
