More than 119 million passengers will fly Ryanair this year. And more than 1.3 million people visit the airline’s website every day. But not many folks get the chance to see what goes on behind the scenes to keep the website humming and the company’s 360 airplanes filled with customers.
So follow along with Ryanair Infrastructure and Operations Manager Declan Costello on this virtual tour through his New Relic dashboards. “We use New Relic for all of our day-to-day monitoring,” Declan explains. “It’s our single source of truth for our website.”
Declan reveals that he uses New Relic every minute of every day, which makes perfect sense, given that the financial success of Ryanair depends on its websites and apps performing well at all times. He’s a particular expert in his use of New Relic Insights and NRQL, the New Relic query language, and is happy to share some useful hints and best practices as well the larger goals behind Ryanair’s New Relic implementation.
Measuring the pulse of the business
“Segments sold” is the top metric Declan’s team keeps its eye on. In fact, it’s the number one metric all business teams at Ryanair are watching closely on a daily basis. For instance, for the commercial yields team, Declan’s team takes the company’s predicted yields and breaks them down into one-minute slices. That data is fed back into New Relic Insights where it’s graphed against actual “seats sold per minute” throughout the day. The commercial yields team can see whether actual sales are doing better or worse than predicted and then respond accordingly with pricing changes.
“That particular dashboard is also used by our executive team,” Declan said. “They’re watching it day in and day out on their mobile phones and one executive has it on his wall.”
Declan is also thrilled about the introduction of the timepicker in New Relic Insights. “Normally we look at our dashboards using a timeframe from 2 hours ago through a few minutes ago,” he said. “But we can use the timepicker to compare all our metrics to each other for other time periods. So, for example, we can see that both our number of seats sold and check-ins dipped at the same time on the same day last week.”
Fighting the screen-scraper battle
Declan’s team uses New Relic to help combat one of its biggest problems: screen scrapers. “People scrape the prices on our site and resell flights on third-party websites,” he said. “We are constantly implementing new anti-screen scraper technologies, but it’s a cat-and-mouse game.”
By using New Relic Insights and leveraging data generated by New Relic APM, Ryanair can visualize when a screen scraper is active on an endpoint and then track it to see if the airline’s anti-screen scraper technology identifies the attack and blocks the access. “For example, we might see that someone is doing 9,000 scrapes a minute against us,” he said. “New Relic shows us when the number of calls to the same endpoint is significantly higher than our historic rate. Then we drill down and see which IP addresses the calls are coming from. When the call volume returns to normal, it tells us that our anti-scraping technology has blocked the attack.”
Tracking payment response time
Declan’s team also closely follows the average response time for payments. “If we see that the payment response time has gone up from 4 seconds to 10 and it stays there, then we can look at other metrics in New Relic Insights to figure out what’s going on,” he explained. “We can tell pretty quickly whether it’s only a certain payment type such as Visa that is slow. Then we can focus on working with the Visa card endpoint.”
Using NRQL, Declan can drill down even further. “We might see that Visa in British pound sterling is the only type of payment that is slow, but Visa in euro is normal. Then we know instantly that it’s a problem with a certain bank and we can call them to let them know,” Declan said.
Monitoring failed logins
Declan’s team also tracks failed logins to various MyRyanair user accounts. “One Friday evening recently, we saw a significant increase in the number of failed login attempts to MyRyanair,” he said. “My first thought was that a cybercriminal was running a script trying to hack into accounts.”
With just a little more detective work, the team correlated the increase in failed logins with a forced upgrade of the Ryanair mobile app, which caused people to have to log back in. “If it hadn’t been our own mobile update that caused it, we would have had to take further action to make sure it wasn’t an attack,” explained Declan.
Identifying payment fraud
Another area that Declan’s team monitors is the payment approval rate. Too many pending customer actions could mean that the fraud technology is catching a greater amount of fraud, or else it’s getting too over-eager and putting too many cards into a pending state. Either way, the fraud department can know instantly that something is potentially amiss and take action.
Ryanair has been using NRQL alerts since the capability was available in beta release, and one thing Declan uses it for is payment tracking. “I love my NRQL alerts,” Declan said. “One example of my NRQL alerts is if the number of approved payments gets too low, if the response time gets too high, or if the number of pendings gets too high, then it starts alerting us.”
Solving low-level problems
One day, said Declan, “We got a call from customer service that they weren’t able to do PayPal transactions. With Insights we were able to drill down and see that PayPal transactions were going through. So we compared it as a time series using NRQL and saw that while we were taking PayPal transactions, we weren’t taking as many as we should be.”
From there, Declan’s team checked the servers. “Very quickly we could see that of the 12 servers that were handling PayPal transactions, two of them weren’t doing nearly as much. Looking further, we figured out that the two servers were using three different IP addresses to talk to PayPal. Two of those three IP addresses were not working, so we were failing to connect to PayPal two-thirds of the time from that one datacenter. So we solved a very low-level infrastructure problem with New Relic Insights in about an hour and a half.”
Our thanks to Declan for sharing these insightful stories with us. To read more about how Ryanair is using New Relic to keep its digital properties flying high, read the full Ryanair customer case study: Ryanair’s Ops Team Relies on New Relic to Deliver Passengers and Performance.
Ryanair’s Declan Costello will be a featured speaker at our upcoming FutureStack London event, which will be held on 24 May, 2017, at Kings Place in London’s Kings Cross district. For the chance to hear Declan and a number of other business and technology leaders talk about the future of business in the digital world, register now to reserve your spot. We hope to see you there!
Note: Event dates, speakers, and schedules are subject to change without notice. Ryanair aircraft image courtesy of Ryanair.