Winter is Coming for CentOS 8

Winter is Coming for CentOS 8—but here is how you can enjoy
your holidays after all.

The server environment is complex and if you’re managing
thousands of Linux servers, the last thing you want is for an
operating system vendor to do something completely unexpected.

That is exactly what Red Hat, the parent company of the CentOS
Project, did when it suddenly announced a curtailment of support for CentOS
8^[1] – sending thousands of
organizations scrambling for an alternative.

In this article, we’ll review what happened with CentOS 8 and
what it means for users who have already upgraded from CentOS
release 7 to release 8. We’ll also look at your alternatives for
replacing CentOS 8.

Finally, we’ll do a review of your other option: choosing
extended support. Extended lifecycle support (ELS) can reduce the
pressure to decide on alternative distribution and it may well be
the most practical route for many CentOS 8 users.

Official support is critical

The difficulties around CentOS 8 involve the sudden withdrawal
of official support. Official support window timeframes matter
because it gives Linux users certainty that they will continue to
receive bug fixes as well as patches for CVEs and security
vulnerabilities that emerge.

A fixed end date for support gives users the ability to plan –
either upgrading ahead of the end date, or migrating workloads to
an alternative if upgrading isn’t a viable option.

While this is an important consideration for people who run a
single CentOS instance and for small teams, official support
windows become critical for those who depend on CentOS to support
large-scale workloads involving big server fleets.

A single user or small team can quickly shift distributions, but
planning for any changes that involve thousands of machines is a
whole different story.

A free Linux distribution – with rock-solid official
support

CentOS had its origins in 2002. The project, a 1:1 fork of Red
Hat Enterprise Linux, went through various changes over time. In
2014, Red Hat announced that it would officially sponsor the CentOS
project – but in doing so, Red Hat took full control of CentOS,
including intellectual assets, and the governing board.

Red Hat invested a lot of effort into the CentOS project, and
CentOS enjoyed a fixed release schedule with equally fixed,
reliable support windows. As of late, the CentOS project was
quoting 10-year maintenance support windows which was fantastic
news for enterprise users who could adopt new releases at a pace
that suited them, with long time frames for planning and
testing.

And, of course, CentOS is entirely free – saving companies
thousands in licensing fees. For example, when CentOS 7 was
released in 2014, users were told that they will continue to enjoy
support through June 2024. With CentOS 8 coming out in September
2019, it gave enterprise users a long time frame to test and switch
to CentOS 8.

Some CentOS 6 and CentOS 7 users moved quickly and adopted
CentOS 8, but these users were in for a surprise.

What changed with CentOS 8?

When CentOS 8 was released, the CentOS project (and by that we
really mean Red Hat) promised that it would continue to support
CentOS 8 for about ten years officially – just like it did for
CentOS 7. The original end of life date for CentOS 8 was
May 31, 2029^[2].

That’s an excellent support window for a free-to-use,
enterprise-grade Linux OS which is also 1:1 binary compatible with
RHEL. It meant that enterprise users could essentially avoid paying
RHEL license fees, while still working with a trusted
distribution.

Unfortunately, the good news ended rather suddenly in December
2020 when Red Hat unexpectedly announced that it will no longer
release CentOS as a stable release at regular intervals, instead
focusing on CentOS Stream – a rolling release model, which is
delivered differently and whose suitability for enterprise
application is still unknown.

Products come and go and a change of direction can be somewhat
understandable, but the real sting in the announcement was that
official support for CentOS 8 will be curtailed by almost eight
years – with end-of-life now on Dec 31, 2021^[3] rather than the
originally promised May 31, 2029.

After that date, the CentOS Project will no longer publish
updates for CentOS 8. Bugs won’t be fixed but, more critically, new
vulnerabilities won’t receive patches. In other words, if a major
flaw in – for example – the Linux kernel emerges, you simply won’t
get an automatic patch for CentOS 8.

That is in contrast to what organizations were originally
promised for CentOS 8 – a matching patch within 72 hours of the
patch being released for RHEL 8, right through the middle of 2029.
It creates an enormous headache for tech teams that must now act
fast to replace CentOS 8.

Why doing nothing isn’t an option

You might think that your workloads are running just fine, and
that there’s no need to update your CentOS 8 instances to apply bug
fixes. Or, that you can simply apply internally coded patches or
other remediation measures should a threat arise.

In reality, the risks of running an unsupported OS are
significant. You can use this calculator^[4]
to estimate the costs and get a rough figure for your particular
infrastructure. We’ve published an in-depth article here^[5], but let’s do a quick
recap of the potential problems you face when your OS is no longer
enjoying official maintenance support.

• Breaking compatibility and reliability. An OS
  is surrounded by other software components and if you fail to
  update your OS with bug fixes, you may find that updates to other
  components break compatibility — you end up with updated software
  and services, but an OS that was never updated with the feature
  change.

• Security risks. This is the big one: if you
  don’t receive regular updates to your OS you will rapidly
  accumulate a growing number of security holes in your workload as
  more and more vulnerabilities get published in public – but never
  fixed on your systems. All it takes is one entry point for a hacker
  to gain entry and potential catastrophe to occur.

• Compliance problems. Compliance requirements
  such as PCI require that systems are patched against
  vulnerabilities within a specific time frame. When your OS is
  unsupported you are at risk of breaching compliance requirements
  which can lead to stiff penalties, the loss of customers – or
  indeed losing the right to do business altogether.

That’s just a brief insight into the potential problems of
running CentOS 8 past the end of this year. It’s an enormous risk
which is no wonder that companies are rushing to try and come up
with alternatives.

The problem with CentOS stream

Red Hat isn’t discontinuing the CentOS Project altogether –
CentOS will continue to exist in the form of CentOS Stream, which
will always be one step ahead of the latest RHEL release. While Red
Hat is suggesting that CentOS Stream is a drop-in replacement,
that’s only true for a limited number of use cases.

Many Linux OS use cases – particularly in the enterprise
environment – depend on stable releases: fixed functionality that
can be tested, and the assurance that nothing of substance will
change until the next release. Indeed, Red Hat’s own CTO has said
that CentOS Stream is not a replacement for
CentOS 8^[6].

The move to the new CentOS Stream may affect the release
stability. It will no longer have exactly the same package versions
as RHEL – in fact, packages will land in CentOS Stream before
making it into a fixed RHEL release. Binary compatibility^[7]
may suffer, and some organizations’ workloads cannot easily
accommodate this.

CentOS Stream would be a perfectly acceptable replacement for
some users – some scientific teams, for example. However, most
large-scale user cases involving more than a handful of machines
will need to examine alternative operating systems – or alternative
support options. And there’s not much time left given CentOS 8 is
end-of-life in just a few months.

How about downgrading to CentOS 7?

In one of the few cases where leaving things to the last minute
has paid off, CentOS 7 users are continuing to enjoy the support
window the Red Hat originally committed to – with CentOS 7
maintenance support set to last until June 30, 2024. That’s a
rather useful two and a half years beyond CentOS 8 support.

So how about going back to CentOS 7 as a temporary measure?
There is, unfortunately, no supported downgrade path back to CentOS
7. Yes, some unsupported solutions are out there, but you’re at
risk of ending up with a system that is in some type of
Frankenstein state – containing elements of both releases. You’re
almost certain to experience problems further down the line.

Taking a look at binary compatible
alternatives

We will divide your alternatives to CentOS 8 into two
categories: distributions that are binary compatible with CentOS 8
(and by consequence RHEL 8), and distributions that are relatively
close in purpose – but that will require more work to adopt. We’re
taking this approach because so many organizations relied on the
1:1 binary compatibility between CentOS 8 and RHEL.

Choosing a distribution that is binary compatible with CentOS 8
implies that your team has relatively minimal work in terms of
switching distributions. In fact, you may be able to switch from
CentOS 8 to an alternative distribution just by running a script –
but, tech teams will still need to double-check that nothing is
broken in the transition. These are your binary compatible
options:

Red Hat Enterprise Linux (RHEL)

We mention RHEL first because, by definition, RHEL 8 is 1:1
binary compatible with CentOS 8. Yes, ordinarily, there is a
licensing fee associated with RHEL, but due to the backlash against
Red Hat’s decisions around CentOS, Red Hat decided to extend the
free version of RHEL.

Red Hat has expanded the free of charge Individual Developer
subscription program to now include workloads that involve up to 16
systems. So, if your workloads involve 16 or fewer CentOS instances
and if you’re certain you won’t require a larger number of
machines, RHEL could be a good choice involving minimal
disruption.

Most enterprise CentOS deployments have far more than 16 active
instances and these workloads will incur a licensing fee.

Oracle Linux

Enterprise users might naturally look towards another free
enterprise alternative – Oracle’s 1:1 binary compatible fork of
RHEL, called Oracle Linux. Oracle claims that Oracle Linux is fully
compatible with CentOS, and anyone who already uses Oracle products
will find the tight integration with Oracle’s other products
helpful.

While Oracle Linux has a proven track record in the enterprise
space, there are some issues around the direction of other products
under the Oracle aegis, like Java, that have come up during the
years, and, arguably, instilled some reluctance when going with the
brand.

AlmaLinux

AlmaLinux OS is a 1:1 binary compatible fork of RHEL – and
therefore binary compatible with CentOS. AlmaLinux is under the
purview of a 501(c)(6) non-profit foundation with a Board of
Directors composed of people from around the industry and the
community, and community adoption has grown steadily over the
months. It already supports most hardware platforms supported by
CentOS, is present on the largest cloud provider’s offers and has
matched all the announced releases dates along the way.

There has been some competition between AlmaLinux OS and Rocky
Linux, which was to be expected since both target the same
audience.

That said, AlmaLinux was faster out of the gate with a
production first release than Rocky Linux and the community
reception has been positive. AlmaLinux has also recently become
available as an OS install on Microsoft’s Azure and offer a set of
RHEL UBI equivalent containers as well.

Rocky Linux

The early CentOS project merged with a project called CAOS
Linux, founded by Gregory Kurtzer in 2002.. After limited
involvement, Kurtzer moved on from CentOS to other projects and was
needless to say unhappy about Red Hat’s announcement and the
changing future of CentOS, so rapidly acted to create a new, binary
compatible fork of RHEL – and called it Rocky Linux.

Rocky Linux is binary compatible with CentOS so it is easy to
switch to. The open source project is, however, currently under
Kurtzer’s full ownership and control although he has made statements^[8]
about opening that up to others. So, again, there can be concerns
that there might be a change of course with Rocky Linux – much the
same as Red Hat did with CentOS.

Other binary compatible alternatives

CentOS users can also look at ClearOS and Springdale Linux, but
in both cases the supporting communities are relatively small.
Springdale Linux is backed by serious institutions though – with
both the Institute for Advanced Study and Princeton University
backing it. While ClearOS has links with HP Enterprise, ClearOS 8
has not yet been released which casts a shadow over the
project.

Scientific Linux isn’t an option as the backers, Fermilab, had
said they won’t release another version beyond release 7 – so
there’s no alternative for CentOS 8 here. For some users, Amazon
Linux could be worth investigating – it’s backed by the tech giant
and is a CentOS-based clone of RHEL, but you can only run it on
Amazon Web Services.

Non binary-compatible distributions to
consider

You may well decide that RHEL and its related distributions do
not offer any unique features – aside from the original advantage
that CentOS is a free RHEL clone. Depending on your workload,
migration may be relatively effortless – but you’d nonetheless need
to prepare and test to a far greater degree compared to migrating
to binary compatible distribution.

One of the most obvious alternatives is also one of the most
established – Canonical’s Ubuntu. It is, of course, derived from
Debian – which means it is some distance away from RHEL and
therefore shifting from CentOS to Ubuntu will be a fairly big
operation.

It all depends on how much of your code is specific to CentOS
and whether you rely on vendors for software or write your own code
internally. Either way, Ubuntu has the necessary track record and
it may well be a sensible option.

There are plenty of other, trusted distributions you could think
about. OpenSUSE, for example, is offered free for use by SUSE Linux
and has a solid reputation, it’s been around for more than 15
years. You could also opt for Debian. However, switching to a new
Linux distribution can be more complicated than it sounds. Some
points you need to watch out for include:

• Monitoring and management systems need to change because the OS
  that supports your workload has changed significantly.
• Development efforts required – both to adjust everyday scripts,
  and to change the code in the applications that run on your
  operating system.
• Coping with different package management mechanisms – RPM on
  CentOS, RHEL and related distributions, PKG on Debian and
  Ubuntu.
• Time consumed and risk associated with the migration process,
  which comes down to a complete system re-install given the
  difference between, say, RHEL-based and Debian-based
  distributions.

In other words, choosing a distribution that’s not in the RHEL
family may involve significantly more work than you intended and
it’s not a decision to be made lightly.

Consider extended support to buy time

At the start of this article we promised you an alternative
route that mitigates the urgency created by Red Hat’s decision.
It’s a simple concept: relying on a third-party to extend
maintenance support for CentOS 8.

A good extended support service will cover you for essential bug
fixes and any emerging vulnerabilities. In other words, if a new
threat emerges that affects CentOS 8 your extended support provider
will roll out a patch to counter the threat.

That means that you remain secure – given that new threats are
always patched – and compliant, given that your workloads do not
accumulate vulnerabilities over time. By consequence, you can carry
on running CentOS 8, buying yourself more time to switch to a new
distribution.

TuxCare’s Extended Lifecycle Support (ELS)
for CentOS 8^[9] essentially continues
the RHEL support commitment. In fact, ELS from TuxCare improves on
what RHEL promised for CentOS – with patches rolled out within two
working days instead of three. TuxCare also has the know-how and
the reputation to deliver – with an established product that’s part
of the CloudLinux product portfolio.

TuxCare has committed to providing extended maintenance support
for CentOS 8 through 2025 – giving you several more years to make a
decision about your CentOS 8 workloads, instead of just four
months. It significantly reduces the pressure on your team.

Act now and secure your CentOS 8 workload

CentOS 8 maintenance support is ending, and it ends soon.
Organizations that still rely on CentOS 8 do not have a huge amount
of time to make a decision about an alternative distribution.

We’ve outlined a couple of distributions that you can
essentially use as drop-in replacements, but given that two of
these are brand new it is understandable that you may want to see
how these distributions pan out before you commit.

If that’s the case, consider signing up for extended support to
buy yourself some more time to decide. However, you must make a
decision of some sort. Not acting is not an option – the risks are
simply too great.

###

This article was written for The Hacker News by Joao
Correia, Technical Evangelist at TuxCare. Correia has many years of
experience in IT systems administration, where he learned the
intricacies needed to keep a company’s stakeholders happy and its
systems secure.