Unsettled by Unison’s Fadeaway from Fedora16 min read

This is in part a rallying cry for packagers, but also a story illustrating how fragile user workflows can be, and how some seemingly inconsequential decisions at the distro level can have disastrous consequences on the ability of individuals to continue running your FLOSS platform.

I’m hoping this 12-minutes read will prove entertaining and insightful to you. This article is the third and (hopefully?!) last part of my Pullitzer-winning “Software upgrade treadmill” série. Go read part two if you haven’t already, I’ll wait. I guarantee you’re going to laugh much harder when reading the beginning of part three below.

---

The year is 2021. After the two-years war against the Machine, and the two-years in the software carbon freezer, I had finally been enjoying two years of blissful efficiency, where I could stay on top of the latest software versions with the Fedora Advanced Pragmatic Geek Workstation Operating System. Certainly I could continue like this unimpeded. As the protagonists said in la Cité de la peur, “Nothing bad can happen to us now 🎶”

But karma had other plans. As I wanted to upgrade from Fedora 33 to Fedora 34 (and newer) in the spring of 2021, I found out that Unison, another mission-critical app I’ve been depending on everyday for the last 15+ years, has been orphaned from Fedora. Déjà-vu, anyone?

Apparently, Unison was not meant to come back into Fedora until someone fixes it upstream, which I assumed to mean this, this and this.

The core issue for downstream packagers was that Unison was very fragile and incompatible with itself across different versions of OCaml, the language it is programmed in. Personally, for my own usecase, I don’t care about compatibility with “other distros’ versions of Unison” or even “across different releases of Fedora”—both are nice to have, but apparently hellish to manage—I am happy enough if I can already be compatible with myself, within the same version of Fedora with all my machines upgraded in lockstep. But when I started drafting this blog post at the end of 2021, I couldn’t even consider that option, because the package no longer existed in Fedora.

In practice, I was once more stranded on an old unsupported version of Fedora (in this case, F33) across all my personal computers, because of a single package that my daily life depends on.

F33 was declared dead and buried at the end of 2021, and as I would no longer be able to receive any updates for it, I spent the better part of the 2022 winter season on a system with wildly outdated packages (I was still on Firefox 95 until recently, for example). Yes, I did hear thousands of infosec voices cry out in terror as I wrote the above.

Here is what sitting in the cockpit of a Fedora battleworkstation feels like when the cord gets pulled again:

I couldn’t be waiting in suspense for such a long time anymore with only the vague hope that maybe someday the situation might come to a resolution. I’ve been there, done that, got the t-shirt and the grey hair to prove it, as you’ve seen in part two of this trilogy. At some point, above-and-beyond loyalty becomes “acharnement thérapeutique“, and you start questioning your life choices.

Back then I wrote this in my journal:

I could survive like that a little longer, but there comes a time where the unsupported, time-frozen Fedora 33 on my machines becomes untenable—too bitrotten or insecure to use—and I don’t know what I will do then. This might, very regretfully, force me to switch away from Fedora—which I had been using every day for over 11 years by now—to another Linux distribution entirely in 2022-2023, and I am really not looking forward to that distro-hopping nightmare, I would really like to avoid that outcome. None of the alternative distros/OSes excite me.

I want to keep using Fedora, but I’m backed against the wall here, and unless someone repackages Unison as a flatpak, or some new folks fixes the issues upstream so that it can be repackaged in Fedora, then I eventually will have to abandon Fedora Workstation as a platform for myself—and doing so for one remaining damned application breaks my heart.

– Me, some months ago

Throughout the winter, I spent countless hours researching and discussing this with potential collaborators (I did do some lobbying to encourage folks to review and merge the existing work too, I think). I believe I have rewritten this blog post at least four times as the situation kept evolving quite a bit during the span of two months this winter:

• Back when I originally drafted this blog post near the end of January 2022, the situation looked quite bleak: there was an attempt at doing an OCaml-agnostic version of Unison in late 2020, but it seemed to have stalled for over a year, so I thought the situation might take years to “naturally resolve” in the then-current state of things;
• Unison’s compatibility issues arguably were both an upstream and downstream problem: if it bites Fedora, it’s likely to bite other distros at some point in time, and the impact of that on users is quite significant. Unison is this kind of non-glorious software that has quietly been used by many people around the world for decades, and I can’t possibly be the only one out there using it regularly (even Debian’s incomplete opt-in statistics show thousands of users);
• Hubert Figuière valiantly made a first attempt at flatpaking OCaml + Unison as a personal challenge, but the effort stalled (I presume because of technical difficulties of some kind). I’m not even sure it is possible for Unison to work in a flatpak, this remains an open theoretical question (see further below).
• Thankfully, sometime during the late winter of 2022, in the upstream Unison project, the code was reviewed, merged and bugfixed, so those three upstream issues (linked above near the start of this article) have been resolved, leading to the highest activity level in years (if not decades), and a new release bundling all those improvements. The ball is now back into the court of the Linux distros.
• Later on, I discovered a third-party repository that provided packages for Fedora (see further below), which would allow me to keep my systems working.

So nowadays we are missing two things: someone to package Unison back into the main Fedora repositories, and ideally someone to potentially package Unison as a flatpak (if that is even possible) in Flathub as well.

---

Help needed to bring back the Unison packages in Fedora

In recent months, I discovered a third-party Unison repository for Fedora by Chris Roadfeldt, where he packages the latest version of Unison. This is the main reason why I’m not running an insecure operating system right now. I asked why this was in a COPR, rather than Fedora’s main repositories, and I was told this:

“I package unison for fedora for my own reasons, which I suspect are like yours, I use it and use it often. I would not be opposed to submitting my edits to the spec file to the fedora main repo, after all it was fork from there years ago.
There is a decent amount of work needed to clean up the package for general consumption and maintainability though, and I am not in a position to do that right now. For the time being I will continue to maintain my copr repo and let others make use of it as they want. Should it be of use in the main fedora repo, the fedora project is welcome to include it.” […]

Looks like we’re already “mostly there”. If you are an experienced Fedora/RPM packager, your help would be very welcome in mainstreaming this (or reviving the previous packages from the main repos). Will you pick up that guitar?

Currently, there is no such thing as a Unison flatpak package. Upstream does not want to maintain or support packages (their policy is to only ship source code), and I don’t even know if it is conceptually possible for a networked system app like unison—which must act as a client and server, commandline and GUI—to work as a flatpak.

The way Unison works is… sorcery. It somehow requires Unison to be installed on both sides (historically with the exact same versions of everything, otherwise things tend to break, but with the 2.52 release this problem may finally be gone), yet it does not run as a system service/daemon, so… how does it behave like a client & server without being a server daemon listening? 🤔 I don’t know how such arcane magic works. But it sure brings up the question: is it actually possible for it this to work while inside Flatpak on both sides? I have no idea. I would love it if someone proves it is possible and publishes it on flathub, thereby solving this interoperability madness once and for all, across all Linux distros (Fedora is not the only one struggling with this, Debian has also frequently been encountering this problem, and therefore, so are all its derivatives)… though arguably, with the new OCaml-version-independent line of Unison releases (2.52.x), this may no longer be an issue (I hope).

---

“Why are you blogging about this instead of fixing it yourself?”

There are couple of reasons. In no particular order:

• I think it’s interesting to raise awareness about these type of struggles users face, and how they can influence the choice of a platform. I was this close to leaving Fedora, and yet I’m someone with inordinate amounts of patience and geekiness; that should tell you something. Furthermore, I would argue that general users shouldn’t have to learn code surgery and packaging (or containers or some other exotic OS mechanisms) to be able to continue using their everyday productivity software. To have mass appeal in the market, Linux-based OSes need to be able to solve everyday productivity problems for users who are not necessarily developers or packagers.
• I don’t have the technical skill required at all; I don’t know OCaml or any of that fancy advanced coding they use in Unison, nor do I have the time/skill/inclination in becoming a packager of any kind. It is really not supposed to be my area of expertise & focus as a management & marketing professional.
• I really don’t have the time for hands-on deep technical wrangling anymore, other than filing bug reports; even just keeping on top of the thousands of bug reports I have filed in recent years (in the GNOME & freedesktop stack) is challenging enough, as the picture below illustrates:

After all, one of the reasons I haven’t blogged much in recent years is that I am trying to build and scale not one, not two, but three businesses. Good bug reports take time and effort, and beyond testing & feedback, I can’t be fixing every piece of software myself at the same time. The reason I have spent much time and effort writing this longform article here is because I find it interesting as a case study on the vulnerability of user workflows to modern operating systems’ fast-paced changes.

---

Syncthing is not solving the same problem as Unison—and never will—as the Syncthing developers say themselves in various discussion/support threads (for example here, here, here, etc.).

I do love and admire Syncthing, it is the best of breed in distributed continuous file synchronization (according to my own casual testing and the many recommendations I’ve heard from people around me). Back in the 2005-2010 era, I would have killed to have this kind of software available to me. Syncthing is, since 2013, fulfilling the promise of the long-lost and buried peer-to-peer version of iFolder (from the Novell Linux desktop era) that I was never able to run except on some of the very early versions of Ubuntu (4.10 to 5.04, maybe?). I wish someone had told me about Syncthing’s existence earlier than 2021! This month, I have successfully deployed it for a friend of mine who was struggling with horrible proprietary slow-synchronizing NAS bullshit.

Syncthing is a technological marvel, and I’d want to use it, but I can’t, at least not for my most important files and application data. My needs have evolved, especially as I want to be able to selectively sync my Liferea data folder, my GTG data folder, my LibreOffice profiles, GNote, SSH configuration files, fonts, critical business documents, and all that while my home partitions/drives are always 99.9% full:

There is no other open-source alternatives to Unison, which is not only peer-to-peer and resources-efficient, but also based on the concept of manual-controlled batches (you can review/approve/override each file/directory being synchronized). I am not the only one out there to have evaluated Syncthing, and gone right back to Unison. It gives you the fine-grained surgical precision you cannot have with other software.

There are many cases where Unison, with its ability to let me override the direction of a sync for any file or folder, has saved my ass, letting me revert mistakes from another computer.

“Sounds like a lot of community management work. Why don’t you just switch distros?”

Pretty much no other distro does exactly what Fedora does (and listing the possible alternative distros here would be beyond the scope of this blog post).

For me, the whole point of using a Linux distribution like Fedora is for it to “Just Work™” and for me to move on with other problems in my life, while still being able to run the latest versions of software so that I am not burdening developers with outdated bugs when reporting issues (so no, unless I decide to stop contributing to GNOME, I cannot really run Debian Stable, Ubuntu LTS, or elementary, sorry!). Under normal circumstances, Fedora provides the perfect “Goldilocks” balance between freshness and stability. If I wanted to constantly break and fix my computer and to mess with packages, I would probably be running Slackware, Gentoo, Arch btw, or whatever übergeek distro du jour, but I don’t want that. I want simple things, and I want the perfect balance that Fedora had struck for me for the past ten years:

• it provides the latest (and vanilla) versions of all software, while still being stable and tested enough to not constantly pull the rug from under my feet; I want the latest GNOME every six months (so that I don’t get yelled at for using ancient software versions when reporting issues) but I want it pre-tested and not shipped on “release day zero” (rolling distros scare the hell out of me in terms of productivity).
• it does not force me to stay on top of 1000-1500 updates per week like you often see in rolling distros (my average uptime on my laptop and desktop workstation is 30 to 45 days on average) nor require ostree and BTRFS to have a semblance of safety (because those two bring problems of their own that I am not ready to accept yet);
• it gives me the ability to wait 1-2 months after each Fedora release before upgrading, so that I can let Fedora early-upgraders (along with Arch and OpenSUSE Tumbleweed users), take the first wave of bugs in their face 😉

---

Conclusion to the Upgrade Trilogy

When I originally wrote the first few drafts of this third article in the winter, the jury was still out on whether I actually would have to switch away from Fedora, because, from a timing perspective, I assuredly would not be meddling with my computers in the winter (you do not mess with your productivity tools during income tax season). I vowed to leave the problem on ice until spring/summer at the earliest, and thought, “Perhaps in the meantime someone will have come up with a solution by then.”, but not knowing if I would still be able to use my favorite operating system six months down the road was a big concern of mine.

Other than writing this educational editorial piece, I did not know what to do while I was sitting again on a ticking timebomb. As a user, I felt trapped and cornered, and that’s speaking as someone who has been around the freedesktop software platform for 19 years and knows how to fix most common Linux desktop usage problems! Imagine what happens when users are not so unreasonably persistent. What is the path forward for users left between a rock and a hard place?

With this trilogy, you now have a poignantly detailed picture of how rough of a ride our FLOSS platforms can be when a seemingly “tiny piece” of the system breaks down. I hope you have enjoyed my writing and found it insightful in some way.

If you’ve read through the whole three parts of this series, I commend you for your dedication to reading my epics. Feel free to share this trilogy or leave a comment so that I can give you a high-five whenever we meet someday.