Why You Should Consider QEMU Live Patching

Sysadmins know what the risks are of running unpatched services.
Given the choice, and unlimited resources, most hardworking
administrators will ensure that all systems and services are
patched consistently.

But things are rarely that simple. Technical resources are
limited, and patching can often be more complicated than it appears
at first glance. Worse, some services are so hidden in the
background, that they just don’t make it onto the list of things to
be patched.

QEMU is one of those services that tend to create difficulties
with patching. It works away in the background and is easy to take
for granted. Plus, patching QEMU involves significant technical and
practical challenges – while requiring enormous resources.

In this article, we’ll address some of the difficulties around
patching QEMU, and point to a solution that takes the toughest bits
out of QEMU patching.

Ignoring QEMU patching is a big risk

You’ll probably know about it if you’re using QEMU – short, of
course, for Quick EMUlator – because QEMU will be delivering
critical virtualization capabilities that support your workloads.
That said, what you might not realize is that just like the host
OS, the virtualized OS, and all your applications, QEMU also needs
to be updated on a regular basis – even though it works in the
background.

It’s not just a scare story. QEMU has been proven to be just as
vulnerable as any other service, library, or component. For
example, in 2015, the virtual floppy controller in QEMU was found
to be vulnerable: it was called the Venom bug^[1], and affected systems
whether the QEMU virtual floppy was in use, or not.

Likewise, in 2019, organizations that use the KVM/QEMU
hypervisor^[2] to run Linux instances
were at the receiving end of a security flaw that put countless
systems at risk. And, just like any other commonly used software,
it’s likely that more flaws will be discovered in QEMU.

In other words, if you don’t patch, your systems will be at
risk. But there’s a problem: when it comes to QEMU, patching isn’t
straightforward because patching QEMU affects the underlying
virtualized workload: while you stop to restart QEMU, the virtual
workload must stop too.

Your options for patching QEMU

Patching a single service on a single system usually isn’t an
issue – assuming you remember to do it – and even patching a single
OS isn’t that hard as you can usually cope with a single restart,
but it is disruptive nonetheless as every application restarts.
Patching a fleet of operating systems is a lot tougher, because it
could imply thousands of restarts and disruption to countless
numbers of apps.

Because QEMU is a virtualization service patching has far bigger
implications than simply patching another application. Patch QEMU
and you have to restart the underlying operating systems that run
on it.

In others, applying a patch to a single service – QEMU – can
lead to the forced restart of thousands of operating systems. It
significantly complicates QEMU patching – and it can mean that tech
teams sometimes delay patching QEMU, trying to justify taking a
risk with vulnerabilities because they view the disruption as too
big.

Patching is a must, however, and there are of course shortcuts
when it comes to updating QEMU – and the right way to do it. Here
are some of your options.

The quick but very risky method

Your simplest, but most disruptive option, is to simply apply a
patch, restart, and see what happens. If it’s just a single
machine, you could be okay – after all, you’ll be aware that you’re
going to need to restart your workload.

However, if you’re managing QEMU across a server fleet, or in
environments where there are external stakeholders evolved, simply
patching and triggering reboots across all the machines will,
without a doubt, lead to many upset people.

A sensible approach

Instead of just restarting, most level-headed sysadmins will go
and add a bit more planning to the above procedure. To start off
with, you’ll notify everyone affected by setting up a planned
maintenance window with scheduled downtime – say, a month in
advance. The problem is, of course, that you’d have to hope you are
not hacked within that month.

However, during the maintenance window, you’ll have an
opportunity to patch without upsetting anyone, permitted a few
hours of no service is tolerated. Once you restart QEMU, all the
virtual machines should restart, and you can inform the
stakeholders that patching is complete.

Nonetheless, you’re likely going to set yourself up for a fair
period of troubleshooting after the restarts, and though you won’t
get anything thrown at you, even planned maintenance windows are
challenging for everyone involved. There are also many scenarios
where planned maintenance that involves actual downtime simply
wouldn’t be acceptable.

Enterprise-grade approach

Some workloads won’t deal well with the disruption caused by
operating system restarts. In enterprise environments, you’ll need
another plan. You’ll need to take a much more involved approach: a
live migration of the QEMU workload.

You can only do this if your workload is already split across
multiple hosts, and where you have high availability activated
across those nodes. You then kick off patching by informing your
stakeholders that a maintenance window will be due, which will
affect performance – but that it shouldn’t affect availability.

Relying on your high availability operation, you migrate the
virtual machines across, then stop QEMU, patch it, and restart it.
After the restart, you migrate the virtual machines back to the
patched QEMU instances.

Done correctly, patching by migration ensures that your QEMU
instances are safely patched without upsetting stakeholders through
real downtime.

The problem with QEMU migration

We’ve talked about three different approaches to patching QEMU,
and the migration route is, without a doubt, the best option for
organizations that rely on QEMU to drive large workloads. But even
this enterprise-grade approach carries risks. You are performing a
very complex procedure that, as all complex procedures, can always
fail.

Some of the things that go wrong include:

• Performance may be significantly degraded during migration –
  which may impact stakeholder and user satisfaction, particularly
  where migration takes longer than expected.
• Coordinating a maintenance window, which is nonetheless
  required due to possible performance disruption, can still be
  challenging and time-consuming – while leading to a degree of
  annoyance for stakeholders.
• During the migration operation, minor network packet loss
  should usually be tolerated – but some workloads can be sensitive
  to this, which can cause significant problems.
• You need to test and verify post-migration – you can’t assume
  that everything has migrated smoothly, and you may need to involve
  stakeholders through this testing process.

Performing QEMU updates through a migration process limits
disruption, but your team nonetheless needs to invest significant
amounts of time in the process. The risk that something goes wrong
remains – and there’s a small risk of catastrophic failure.

So, while it’s unlikely your stakeholders will see significant
disruption, your team will need to do careful planning. Finally,
it’s worth considering that any adverse outcome of the migration
process – small as the risk may be – will reflect negatively on you
and your team.

Live patching as an alternative

In the past, patching always depended on a stop, patch, restart
process. Yes, migration helps by ensuring the instances that
require restarting are available. But a fresher approach has become
increasingly common: patching on the fly, without restarting the
software that’s being patched.

Called live patching, this approach significantly simplifies the
patching process. Instead of requiring a restart, live patching
updates your server or the service you need patching on the go.
That’s the case for QEMU live patching too, where you can now
install the latest patches for QEMU – without setting up a
maintenance window, nor the need to execute and plan migration.

That’s why QEMUCare^[3], from TuxCare^[4], is a game-changer for
teams that run workloads on QEMU. QEMUCare doesn’t just make the
update and migration process easier – it takes it away completely.
Your QEMU/KVM instances are patched instantly with no impact on the
underlying virtual machine.

Choosing the live patching route brings a whole range of
advantages:

• Consistent patching. A good live patching
  solution such as QEMUCare will automatically detect the release of
  a new patch and initiate the patching process. Your team doesn’t
  even need to monitor for patch releases: QEMUCare just takes care
  of it. That means that your team patches more consistently –
  reducing the risk that your QEMU instances are vulnerable to a new
  exploit.
• Happy stakeholders. Because QEMUCare works in
  the background, automatically patching without rebooting QEMU, your
  stakeholders – including internal users and your customers or
  clients – won’t even know that you’re performing patching. It all
  happens seamlessly without the need for planned maintenance
  windows.
• Eliminates labor hours. While you have the
  option of trying to take a shortcut, the enterprise-grade,
  migration-driven process for patching we described before is your
  only realistic choice. It is very labor-intensive, however,
  consuming lots of hours from your team – whereas QEMUCare^[5]
  consumes almost zero hours from your team.
• Minimizes risk of error. Because you don’t
  have to migrate your workloads manually there is less risk that
  patching QEMU will cause you significant problems. There are no
  migration glitches or network errors to worry about – and you and
  your team members don’t need to worry about your jobs.

Clearly, live patching greatly simplifies the process of keeping
your QEMU instances up to date: it happens automatically, you don’t
need to worry about anything going wrong – and you don’t need to
invest a lot of time to get it done.

QEMU patching is essential – and live patching makes it much
easier

QEMU may be quietly doing its job in the background, but you
can’t ignore it from a cybersecurity perspective.

You must patch QEMU, but it’s understandable that your team may
be daunted by the prospect.

While thorough planning and a maintenance window will get you
there, live patching just makes it so much easier – you can patch
more frequently, and with less effort. So, if you’re dependent on
QEMU for your workload, consider how live patching from TuxCare can
benefit your team.