The post-2020 Linux server landscape metamorphosis7 min read

This is “part one” of a three-part blog post on the challenges of keeping up with the “software updates treadmill” in the land of Linux. The next two parts are going to be about the Linux desktop. This first part focuses on the server side and will require about 5 minutes to read.

It used to be that you could leisurely deploy a L.A.M.P. server, and stop caring about it for years because PHP’s releases, and the dependency changes in web applications, were happening really slowly. Not so anymore. With the 7.x and 8.x series, PHP has considerably sped up its releasing cadence, and shortened the shelf life of releases. I’ve seen a drastic shift happen in the policies of web application developers, including Matomo (née Piwik) and Kanboard. Even WordPress, one of the most conservative behemoths of the industry (understandable, given that they power roughly half of the websites in the world), requires PHP 7.4 and no longer runs on PHP 5.x.

“Just put everything in containers and continous-deploy all that shit!” I hear you say, “It’s the future!” But I’m not a sysadmin, I’m not day-in-day-out into that crap, and the only reason I run a dedicated server machine in the office is because Matomo doesn’t scale well on shared hosting and their SaaS prices are quite expensive for an individual when you don’t like being artificially capped to a certain number of visitors per month, and, y’know, “How hard can it be, really?”… but I am happiest when I never have to touch/upgrade that server and don’t have to learn rocket science to deploy something. I understand now how infrastructure work would eventually turn you into a Bastard Operator from Hell™.

Circa 2014, I deployed CentOS 7 on my personal server to be able to run Matomo with better performance, because the Pitivi website had a lot of visitors (which is useful to derive knowledge such as “what screen resolutions do people actually use and what can we afford for our UI’s design?”) and its Matomo database weighted multiple gigabytes.

Fast forward a couple of years, and I’ve fallen behind on Matomo updates because, in part, of newer PHP requirements needing me to resort to third-party repositories to get a recent-enough version of PHP to run it. But I eventually did, and it worked, for a time.

Then in late 2019, CentOS 8 came out, so I switched to that, but you can’t just upgrade from one major CentOS version to another, you have to clean install, so it had to wait until early 2020 when I mustered enough courage and enrolled a friendly geeky neighbor to help me out with that yak shave. “Good, I’ll be set for another half-decade now”, I thought.

Then, at the end of 2020, CentOS 8’s support cycle got shortened with an EoL date set to the end of 2021, and the public was told to migrate to RHEL or CentOS Stream instead, and there was much discontent on the Interwebs about that, to say the least. But fair enough, I migrated to CentOS Stream, because I’m not running a mission-critical server powering the stock exchange; CentOS Stream was the easiest path forward, a couple commands and you’re done. Cool. I can live with that.

My respite was short lived however, and my stress levels are rising again as—since the announcement of CentOS Stream 9—I am realizing there may be no upgrade path planned for the various releases of CentOS Stream. Or at least, the lack of such documentation in the announcement, the worrysome comments there (ctrl+F “upgrade”), the fact that my request for clarification remained unanswered, and a redhatter’s comment in this Reddit thread saying that there is no upgrade path (and none is planned), are all factors that do not inspire me much confidence.

I love Red Hat (and the majority of my FLOSS friends work there), have tremendous admiration for their work, and wish them the best. While the creation of CentOS Stream may be a great move from a development & maintenance process standpoint—I can see the appeal, really—the way this situation was handled is not exactly the way I would have handled it from a PR & marketing standpoint (Red Hat unfortunately got a ton of flak and eroded some of its trust in the process), and realizing now that there is still no upgrade path between the short-lifespan* CentOS Stream releases, really discourages folks like from being able to casually use this platform. I know I’m not the only one in this situation.

I would prefer to remain on CentOS for my web infrastructure needs, but this may very reluctantly make me question my allegiances and consider Debian Stable, because I can’t deal with this kind of uncertainty and tedious “clean install” work a third time in such short timespans. And frankly, at this point, the prospect of going to Rocky Linux does not excite me particularly; I might encounter similar worries about the platform’s future or struggles when it comes to upgrades, so if push comes to shove, I’d be more inclined to hedge my bets with Debian, because it’s the one constant in the Linux landscape: Debian was there in the beginning in 1993, and it can’t be killed—it’s reasonable to say now that it will always be there until the end of times, free of ownership influence. And at least the damned thing can be routinely upgraded from one release to the next. Life is too short to be spent clean-installing servers.

With all that said, it remains to be seen whether Red Hat’s community entreprise OS strategy will be a net benefit from a business perspective, beyond engineering considerations. Of course, I realize I am not the target customer when it comes to enterprise scenarios, nor am I owed anything for free; but word of mouth and goodwill are very fragile things, and while I personally keep a positive outlook on Red Hat, not everyone around me does. The price Red Hat might have paid in potential brand damage, burning a lot of goodwill in exchange for increased development process efficiency and potentially a small sales uptick, in the resonating words of Dormin, “may be heavy indeed.”

---

Erratum: CentOS Stream 8 has a multi-year lifecycle

*: it was now pointed out to me, after this blog post came out, that CentOS Stream 8 isn’t supposed to end with CentOS 8’s Q4 2021 EoL date. This was not self-evident to me, nor was it (in my opinion) documented extremely clearly in the original CentOS Stream announcement, the FAQ, or the EoL information page, which did not specify dates for CentOS Stream 8 specifically (so casual onlookers like me could be led into thinking it has a short lifespan and that you would need to continuously upgrade from one Stream release to the next—which was an acceptable deal to me, until I realized there is no upgrade path!), and the fact that Stream releases have multi-year shelf lives was kinda buried under FAQ entry #6’s inconspicuous title, “Will there be separate/parallel/simultaneous streams for 8, 9, 10, etc?”—so those pages could certainly benefit from some clarifications to clear up any such misunderstandings in the eyes of the public.

I now know my CentOS Stream 8 server probably won’t stop being supported as of Q4 2021 after all, which is a relief; time will tell whether Stream remains an attractive proposition (if an upgrade path opens up in time for its EoL) or whether I may have to look elsewhere.