By now you’ve most likely heard about the Spectre and Meltdown vulnerabilities, a pair of deep-seated bugs that could potentially expose kernel memory in many computers’ central processing units (CPUs). It’s been reported that almost every computer manufactured in the last few years is affected, and we all need to act quickly to update or patch our systems.
A lot has been written about the technical details of Meltdown and Spectre, but I want to focus on how I used New Relic Infrastructure to find hosts that needed to be patched in my personal Amazon Web Services (AWS) infrastructure. You can use similar techniques to identify hosts to be patched on other cloud vendors or in on-premise infrastructures.
Do your research before trying to patch Spectre and Meltdown
According to the AWS Processor Speculative Execution Research Disclosure bulletin, which discloses the vulnerabilities, AWS has already patched the parts of the systems it was required to, as guaranteed in the AWS Shared Responsibility Model. This agreement states that AWS is “responsible for protecting the infrastructure that runs all of the services offered in the AWS Cloud,” and customers are responsible for maintaining “guest operating systems (including updates and security patches) … installed by the customer on the instances.”
So following the guidance in the bulletin, AWS suggested I update my host instances. Since I use Amazon Linux, I found I needed to update the kernel package on each instance.
Find the instances to update
To find all the instances that didn’t have the correct kernel package, I used host filtering in New Relic Infrastructure.
Note: You can also use the Infrastructure Inventory page to find instances, based on variants, that need to be patched. For that approach, I suggest you refer to the documentation on searching your infrastructure using the Infrastructure Inventory page.
To filter hosts by OS and kernel version:
- In New Relic, select the Infrastructure tab.
- Click FILTER HOSTS.
- Click operatingSystem and select the OS you want to filter for.
- Click kernelVersion and select any kernels that don’t match the new, patched version.
At this point, New Relic will show you a list of hosts instances for which you’ll need to update the kernel package. It’s that easy!
Well, it’s that easy to find the instances you need to update; be sure to follow the proper procedures for updating your operating system, and review your company’s change control processes before you patch or update your instances.
Before you patch, note the time
It’s always a good practice to track as much detail as you can when mitigating vulnerabilities. Before you update your instances, make a note of the date and time. This way you can use New Relic Infrastructure to view your CPU usage before the update and compare it to usage after you run the patches.
Based on the filter you created in the previous section, you have the instances you need to benchmark.
Here’s an example of what my instances looked like before I patched them. I’m measuring the percentage of CPU, CPU User, CPU system, and memory that’s being used on each host, as they are key metrics to watch for performance during a patch.
Get in touch and share your findings
This latest industry panic has been 20 years in the making, but with New Relic it should take just a few minutes for you to find the instances you need to fix and understand if the fix impacts your infrastructure utilization or application performance.
We’d love to hear from you about any performance and utilization changes you’ve experienced dealing with Spectre and Meltdown. Share your insights with us in the New Relic community forum, or feel free to direct message us on Twitter @newrelic.