The country’s biggest municipal power system has wasted no time in implementing patches to the Meltdown and Spectre processor vulnerabilities.
“We have been issued patches from many of our vendors (while still awaiting from others) and have started installing them,” Robert Wong, executive vice-president and CIO of Toronto Hydro, said in an email on Friday. “Given reports that the kernel patches may impact system performance from five to 30 per cent, we are preparing to provision more CPU’s to our servers should the need materializes.”
He advises CISOs to immediately contact all of their IT vendors to see if patches are needed for servers, endpoints, databases, middleware and networking equipment. These patches should be run through test and development environments to assess possible performance impacts which may need to be compensated for before being put into production.
Security teams should also be told to watch for potential signs of exploits against these CPU flaws, Wong added.
Finally, all employees should be warned to patch their personal devices.
“There is a significant amount of confusion,” among infosec pros and the public, said Robert Beggs, CEO of Digital Defence, a Toronto security solution provider. “Overall, the Windows users seem to accept the issue as, ‘another day, another patch.’ …. the Apple users seem to be scrambling to find out how deeply they are affected.”
Meltdown(CVE-2017-5754) affects only Intel CPUs, while Spectre (CVE-2017-5753 and CVE-2017- ) affects nearly all recently make processors.
As more information spreads about the processor vulnerabilities CISOs face two questions: How fast do I need to remediate, and how much – if any – of a performance hit will my systems have to absorb?
On the first there are two schools of thought:
1 –Take your time, there are worse threats out there. That’s the advice of Johannes Ullrich, CTO of the SANS Institute’s Internet Storm Center.
“I would expect on Patch Tuesday (tomorrow, from Microsoft) there are patches coming out that are more important that this one,” he said in an interview. “I’m guessing there’s usually your Microsoft Office code execution flaws (in the list of patches). Those are the kind of flaws that are responsible for the vast majority of exploits we’re seeing right now.”
Meltdown/Spectre are “not a vulnerability that is going to be used today to take down the infrastructure, he said. “In order to take advantage of the vulnerabilities [attackers] need to be on your system, for the most part. There’s probably a lot of other things you should do before you apply this patch. This is not a patch you need to rush out … Test it, roll it out as (part of) your normal patch procedure. But this is not a panic thing like WannaCry, where you’re going to be hit tomorrow if you don’t patch today.”
2 –Patch fast, because now that the word is out threat actors will try to exploit the vulnerabilities as quick as they can. That’s the advice of Amir Belkhelladi, partner in Deloitte’s risk advisory service for Eastern Canada.
“If our clients don’t follow that advice and patch as quickly as possible, what’s going to happen is the hacking community will start building up ways to exploit those weaknesses … So it’s really a race against time to apply patches.”
Still, standard patch management procedures should be followed, he stressed, including test before deployment.
If patches for a system are needed but haven’t been issued, ask yourself if the system is critical. Consider the business risk of running it, he said. If it’s a critical system can data be transferred to one that has been patched or immune to attack?
“If everyone can patch this over the next two or three weeks they’ll be a lot safer than if they sit on it for a year or so until someone [issues] a very good hacking toolkit.”
“The key is to be able to understand the risk and then manage it quickly.”
Bryan Pollitt, vice-president of professional services at Information System Architects (ISA), a Toronto-based security solution provider, is also in this camp.
“The normal patch cycle involves a level of testing and validation that in this particular case is likely not going to occur, at least to the same degree,” he said in an interview. “This would be an emergency patch.” Mature infosec teams have a procedure for quickly testing and implementing patches quickly – while ensuring the fix doesn’t make things worse — when necessary.
“The first thing is not to panic,” he stressed. It’s important for the CISO to get detailed information and advice from vendors or consultants to on what to do in a timely way. “It’s reasonable to conclude the bad guys are working on exploiting the vulnerability that’s now very public. For that reason organizations interested in protecting their brand, their data and their privacy must conclude deploying the patch as fast as humanly possible is a good idea – but that needs to be matched with a prudent approach to making sure the patch does not do any harm … and does not impact performance beyond a reasonable degree.”
“Although this is a very significant vulnerability and exploiting it would present an attacker with an extremely wide attack surface, there are patches available, and in most cases they have been tested with the security software that would be needed to run on the system. But it’s an opportunity for organizations to respond in an orderly way.”
“The lesson is its obviously beneficial for organizations of all sizes and scope to be as prepared as possible to have incident response plans in place, to have protocols in place, so they can respond in an orderly fashion to an emerging threat or a recently announced vulnerability so they are able to respond accordingly.”
At least one expert has been quoted as saying that because Spectre patches require mitigation techniques that don’t exist the threat won’t be erased quickly. Affected software vendors need to update their compiler infrastructure and recompile their products for patches before releasing updates — and, of course, users have to install the fixes.“That’s quite the pipeline in order to address just one vulnerability with a massive window of opportunity for nefarious actors to cause mischief,” one expert said.
As for possible performance hits, Ullrich said systems most affected are those that do the most reading and writing to disk, such as database application. That, he added, include Web applications like Salesforce and WordPress.
”I heard [Friday] morning that a PeopleSoft shop put it [the Microsoft update] in. They haven’t had any issues with it.” However, he admitted he isn’t sure if that organization had any performance issues. “If your system isn’t utilized very heavily you’re probably not going to feel it that much,” he said.
Pollitt said ISA’s has tested Microsoft’s patch on a system “and have not noticed a significant degredation in performance.” But, he added, a system’s specifications can be altered, so it’s reasonable to assume some performance hit is likely.
Deloitte’s Belkhelladi said the question should be, do you want to run faster or safer? “If your goal is to protect a system, that should take priority over performance … Your priority should be to run a business in proportion to the level of risk you’re willing to take.”
Red Hat has said in some cases a performance hit of up to 19 per cent has been seen in tests on Red Hat Enterprise Linux.
Intel says the performance impact of any updates is “highly workload-dependent and, for the average computer user, should not be significant and will be mitigated over time. While on some discrete workloads the performance impact from the software updates may initially be higher, additional post-deployment identification, testing and improvement of the software updates should mitigate that impact.”
It quotes three major vendors with issuing the following statements:
Microsoft: “The majority of Azure customers should not see a noticeable performance impact with this update. We’ve worked to optimize the CPU and disk I/O path and are not seeing noticeable performance impact after the fix has been applied.”
Amazon: “We have not observed meaningful performance impact for the overwhelming majority of EC2 workloads.”
Google: “On most of our workloads, including our cloud infrastructure, we see negligible impact on performance.”
Apple: “Our testing with public benchmarks has shown that the changes in the December 2017 updates resulted in no measurable reduction in the performance of macOS and iOS as measured by the GeekBench 4 benchmark, or in common Web browsing benchmarks such as Speedometer, JetStream, and ARES-6.”
Meanwhile this morning a European threat researcher notes in a column that a in thread on answers.microsoft.com many users claim that Microsoft’s Security Update for Windows KB4056892 bricks some AMD-powered PCs with Athlon processors.