With more than 80% of our members using digital banking through either the mobile app or Navy Federal Online, the importance of making sure those applications and websites are running perfectly, and that we have full observability into that data right away, cannot be overstated.”

BRETT SIDEBOTTOM, MANAGER CYBERSECURITY TECHNOLOGY ENGINEERING, NFCU

Great organizations realize that creating a first-class customer experience isn’t just the responsibility of the folks on the front lines, or with the folks that speak directly to customers on a daily basis. It’s also the job of teams that operate behind the scenes maintaining infrastructure, digital experience, and security, helping to reinforce our customers’ continued confidence and satisfaction. One could argue that maintaining that confidence is even more important in the financial services industry where technology decisions can impact a customer’s financial future.

Maintaining customer confidence and prioritizing customer experience is extremely important to us here at Navy Federal Credit Union. For those unfamiliar with credit unions in the United States, credit unions are member-owned, not-for-profit financial institutions. Since Navy Federal is the largest credit union in the U.S. with more than 14 million members and $180 billion in assets, we are comparable in size to some of the largest global financial institutions. Navy Federal’s members are U.S. military active duty, veterans, and their families, so we take our responsibility to our members very seriously.

After some significant updates to both our mobile- and browser-based infrastructure, we realized that we did not know enough fast enough to quickly identify problems that could occur anywhere from origin to edge if there was an incident or user experience degradation. Like many enterprise organizations in highly regulated industries, our data environment is a complex one. We have data coming from SIEM connectors, coupled with a variety of other observability data, and traditionally, it has been challenging to connect all that data. For example, we need to correlate synthetic transactions with security events to get the most accurate picture and identify issues.

But just being able to identify a problem right away is only half of the issue. We also needed better incident response capabilities. When a problem arises, such as our login authorization call goes down or if we see server errors, the mitigation process involves heavy lifting and significant time from our team. We would spin up an incident bridge, confirm the issue, and then go through dozens of validation checks to authenticate the situation had improved. About 20 people would be on that call from a number of global regions, each logging in and doing individual tests. Ultimately, the timeliness and fidelity of that data was never great. We had to find a better way, all while making sure the solution was also cost effective.

We knew one option was to build an observability service in house, which involves building our own ELK stack and pulling in cloud monitoring data. The project would require a lot of full-time employee hours to build it and then after setup, we would need to manage it, and control the cost of ingest, storage, retention, and compute. We’d have to commission server space, log storage, and ingest all the data. It would certainly require hiring additional team members, which is time-consuming and expensive.

When organizations talk about building their own solutions and compare that option to outside providers, quite often, they assume that their homegrown build will work. But success is not guaranteed, so there is a great deal of risk with that strategy. A number of groups at Navy Federal felt that this initiative was incredibly important so we owed it to the stakeholders to make sure we mitigated as much risk in this project as possible.

Since Hydrolix provides a managed service, we could spin up a POC within just a couple of weeks for the evaluation process. It was an easy white glove handoff, with Hydrolix managing the configurations, dashboards, and data ingest so we didn’t have to do much. Best of all, there was no risk to the credit union. The experience was as good as any product that I’ve used.

The POC gave us exactly what we were missing: near real-time data observability across our large and geographically dispersed member base. It was easier, faster, and incredibly successful to use the Hydrolix solution as a service, and the cost was very reasonable, especially when compared to any of the big data players out there.

The ability to see the full picture and health of everything across various functions has significantly shortened our time to resolution from hours or days to seconds and minutes.”

BRETT SIDEBOTTOM, MANAGER CYBERSECURITY TECHNOLOGY ENGINEERING, NFCU

Having complete performance and availability data is eye-opening. It means we better understand our member experience, which also means we can measure and continually improve it. For example, if we have a release across our digital banking platform, we can use Hydrolix as the observability solution to evaluate if it’s a good or bad change. Their dashboards give us information so quickly that if a problem arises, we can identify and roll it back to a known good right away. The ability to see the full picture and health of everything across various functions has significantly shortened our time to resolution from hours or days to seconds and minutes.

The impact of the Hydrolix solution doesn’t stop with my team. Everyone from cloud network engineering to cloud platform engineering, from API engineering to incident management, from our digital banking team all the way up to SVPs and even our C-suite, have seen the impact of this data and the value that we get from having it in real time. It’s a solution that has a lot of fans and quickly became Navy Federal’s source of truth.

Without Hydrolix, Navy Federal would be back in the dark ages, and our team would very quickly see the impact of a lack of data. It’s how we know how we are doing at any given point in time. To lose that data would really handcuff our ability to understand our member experience and to react to changes and problems across the platform.

Written by Brett Sidebottom, Manager of Cybersecurity Technology Engineering, Navy Federal Credit Union.

View All Customer Stories