In my free time, I try to help other open source projects get rid of accumulated weight from the years. At GUADEC 2012, I told the Epiphany devs that the 3.x series (with its new design direction/vision and the gradually improving WebKitGTK backend) would be the perfect opportunity to close massive amounts of old bug reports. Indeed, there were nearly a thousand of them open. This week-end, I closed about 150 of them. I’m not even nearly done; this will take time and patience, especially since I want to do the same thing with Empathy, which also expressed interest in a radical round of cleanup. What I’d like to convince you of is the need to reduce our “excess inventory”. Read on.
Although I’m superhuman and can shoot down hundreds of bugs with my laser eyes and uncanny searching abilities, I need some help here. If we had a concerted SWAT team of cleaners/triagers, each closing a dozen bugs every day (focusing together on one specific project at a time), we would get the whole situation under control in no time. Start small but regularly: unless you’re feeling motivated in doing batches/sprints of 50-100 in a row, start by cleaning up a dozen of them per day (and then force yourself to stop). This is a great way to contribute to GNOME if you’re a power-user who doesn’t code.
Some general principles I would like to share in this blog post:
- Move fast and avoid staying in limbo. NEEDINFO everything you’re unsure about. In six months, with a special search query, close any remaining NEEDINFO that has not been clearly answered (if the GNOME bugsquad doesn’t do it for you for free). For example, here’s a search query showing all NEEDINFO bugs that haven’t changed in the last three months.
- Be polite but firm. If you know the project’s vision, show some courage and learn to say no or “patch or it won’t happen”. If you lack experience and have doubts, ask the devs for a “yes/no” decision on a given bug. You’ll learn a lot that way.
- Do regular cleanup sessions every few months or when undergoing a significant technological or ideological change. Know the “big projects” on the roadmap and seize opportunities for cleanups (ex: old untouched bugs, core engine changes, UI redesigns and focus changes, …).
These principles stem from my personal experience, from “lean” manufacturing and the theory of constraints, from the “Getting Things Done” methodology, and from interesting articles such as Havoc Pennington’s or Joel Spolsky’s. The 2nd half of Havoc’s “Free software UI” article is mandatory reading to understand GNOME’s development philosophy of the last decade. Joel’s “Software Inventory” article has some fascinating/provocating ideas when it comes to bug tracking (but really, that’s just lean manufacturing applied to software development):
“[…] the desire never to miss any bug report leads to bug bankrupcy, where you wake up one day and discover that there are 3000 open bugs in the database, some of which are so old they may not apply any more, some of which can never be reproduced, and most of which are not even worth fixing because they’re so tiny. When you look closely you realize that months or years of work has gone into preparing those bug reports, and you ask yourself, how could we have 3000 bugs in the database while our product is delightful and customers love it and use it every day?
[…] stop and fix bugs until you feel like you’re fixing stupid bugs. Then close as “won’t fix” everything left in the bug database. Don’t worry, the severe bugs will come back.”
This is what this blog post is all about. Reducing our excess inventory.
Towards that goal, here are some bug searching tips:
- Search for all bugs reported on [insert your deprecated technology here] (in Epiphany’s case, the Gecko/XULRunner backend). If they still affect the new backend, bump version numbers and change components as required. Otherwise, if they are not easily reproducible, kill them all.
- Search and verify all bugs older than 400 days, 800 days. Chances are they are fixed, obsolete, out of scope, duplicates, or crack (requesting features from the 80’s that 0.1% of users care about).
- Developers/maintainers: search for all unreviewed patches. Forward-port and merge yourself, or ask the reporter to do it and NEEDINFO/close in the meantime. The longer you let a patch sit there (even if it’s not 100% perfect), the harder it will be to manage and the more likely it is to be completely obsolete/wasted.
In the case of Epiphany, I added links in the development page (archived version) for sample search queries for the items above. You can adapt them for most projects.
The art of saying no
When faced with a bug report, you have three choices, especially when it comes to features:
- “Yes”. Only if it fits the project vision and you care enough about this feature to implement it yourself in the foreseeable future.
- “No for now, yes if you provide a patch”. Until then, close the bug report, or if you kinda care about the feature, confirm it, tag it “gnome-love” and possibly lower its priority.
- “No, out of scope. Sorry”. Be respectful and polite (you can’t implement everything out there due to insufficient manpower or because feature X would go against the project vision/intended UX; a canned message for that helps a lot). People will respect your decision more if they know your constraints and why you have to say no.
TL;DR: uncontrolled feature requests are vicious beasts that will eventually cloud your vision and suck your blood. Keep them at a safe distance.
Before you start flaming, keep in mind the context in which I’m saying this: long-term maintainability, a clearly defined scope, reasonable goals, a lean and agile approach, etc. I’m not saying that suggestions for enhancements/features are not valuable (they are, very much), but they need to be carefully and realistically evaluated.
Decisions need to be taken
Indecision is paralysis. Leaving a 7 years old bug report open because “nobody has made a decision” is not going to help developers attain a lean process. In bug 167592 for instance, I politely request the devs or design team to say a final yes or no, not a “maybe, I don’t know”. If we don’t decide now, we never will.
This approach may seem harsh to many (especially if you’ve never done long-term software maintenance and development), but it is the only sustainable way forward. Remember the goal: adherence to a clear vision, minimal “inventory” and achieving a lean/agile project flow.
Additional tips for bug triagers
- Retitle/rename bugs to be clearer, verbose and using common words. This makes them easier to find (preventing duplicates) and makes it easy to know what they are about at a glance.
- The “gnome-love” keyword to identify items that would be easily implemented by potential new contributors is a very important keyword.
- The “ui-review” keyword is your friend. We should start using it systematically whenever we’d like the GNOME design team to advise on a decision to be taken.
- Canned replies are very handy tools in your arsenal. Do take time to prepare some custom stock responses for your project’s particular needs, of course.
- If you’re really crazy about a particular project, you can subscribe to all of its bug mail (see the “About the -maint aliases” section on this page), after your big cleanup, to keep an eye on things in the future.
See the GNOME bugsquad page for more tips.
Which projects to work on?
We can look at the top 15 GNOME Bugzilla modules:
- GTK+, GStreamer and Evolution are of course huge scary beasts that we cannot concern ourselves with for the time being (unless some of you want to make some special triaging hackfest for them?).
- Let’s applaud the efforts on Evolution, though: I remember it having 5000 open bug reports only a few years ago. It can be said that the situation is gradually getting under control there, though the recent de-involvement of SuSE is quite worrying.
- What remains? Core apps we use everyday: GNOME Shell, Rhythmbox (or Banshee, if you use that), Epiphany and Nautilus. In all cases, they are certainly full of old reports that are not reproducible anymore (or fixed), and in Epiphany and Nautilus’ cases, many reports that don’t fit their new vision anymore.
- However, this doesn’t give us a truly complete picture, because this report ignores feature requests. If you count the feature requests, for example, Empathy would probably show up here. It’s a good start though.
There is a big opportunity for cleanup here and we need a larger bug squad, made of users who know enough about the software they use (and the project vision) to help developers keep their sanity and direction. Talk to your friendly developers beforehand, but I’m pretty damn sure they will be delighted to have your help to bring back clarity.
Of course, this mission does not rest solely upon the shoulders of the bug squad. Developers are the best-positioned people to know when a bug is obsolete or not, and it is also your responsibility, as a software maintainer, to periodically go through “old unchanged bug reports” (on a lazy afternoon, once or twice a year, depending on your release cycle). This is what I do, for example, when a new Pitivi release comes out, and this is why the Pitivi bug tracker is a system that is kept in order, a system that I can trust and something I can actually refer to instead of something I would avoid.
2022 update (10 years later): While GNOME has migrated from Bugzilla to GitLab and many of the example search query links here no longer work, I am quite happy with the fact that Epiphany (GNOME Web) has managed to maintain its list of issues under 300, Nautilus now maintains itself under 400 issues, and Evolution’s UI list of issues is also typically under 400 tickets nowadays, instead of 5000+. Various other GNOME app projects seem to have bug report inventories that are under better control, though it typically is, as you may guess, the result of a constant effort to apply the recommendations of this article.
You can check out my main website, find me on Twitter or Mastodon or subscribe to my YouTube channel.
Comments
20 responses to “Reducing our core apps' software inventory”
This is a great post. Bookmarked.
Wow, way to motivate. I didn’t even realize the situation was so messy.
Ah, so that’s why there was suddenly activity on the Epiphany bugs. 🙂
Canned replies: If you do use a canned reply, please remember there’s a human in the other end of the report you are about to close. Perhaps spend the extra 30 seconds and personalize it just a little bit so you don’t sound like a robot. Nobody likes to have something they invested time in brushed aside by a robotic drone.
I totally agree. It’s not helping anyone to have 500+ bugs with no activity for years. I’ve often wanted to prepare a polite message that can be summarized as “please verify this is still happening and reopen if it does” then mass-close all bugs with no activity since ~1year.
I don’t think we should be afraid of closing bugs, the important one will surely be reopened quickly.
“Let’s applaud the efforts on Evolution, though: I remember it having 5000 open bug reports only a few years ago.”
Andre Klapper deserves most of the credit for that. He spent weeks slogging through Evolution’s old bug reports and ruthlessly closing or NEEDINFO’ing them.
One of the problems with agressively saying “no” in bug reports is the fact that, at least in GNOME’s Bugzilla, bug reporters are able to reopen their own bugs. So in a lot of cases, even if I try to be polite, the reporter will feel insulted and immediately reopen the bug. At that point I leave the bug to rot and get back to work rather than continue that bickering. And that just adds to the bug count.
If it wasn’t for the Starship Trooper pictures I would agree, very well written blog post, albeit a bit hard to digest due to the extra length.
Back to the pictures: On the surface, Heinlein wrote a book about killing bugs. I get the joke, OK. But the book (similar with the movie) is not about the bugs, it’s about those killing the bugs. But read on here http://en.wikipedia.org/wiki/Starship_Troopers#Allegations_of_fascism and realize why Starship Troopers pictures in blog post are almost always misleading, unless you write about Starship Troopers.
It’s a stupid cargo cult that people believe totally unrelated but somehow “witty” roflcopter pictures make a blog post easier to read.
A blog post becomes more readable by making the text more readable. Nothing else. Yours was very readable. Why add unrelated & distracting pictures that forces people like me to scroll more? Or to waste more bandwidth on my mobile device?
@Matthew: Bug triage is not the job of the core developer, IMO. Others can do that job, and you can keep coding.
Great post!
There’s nothing wrong in closing a bug that has been open and unattended for a long time (>6 months). If the bug report is incomplete and the developer can’t reproduce the issue, then it’s very likely that it is the end of life of that report.
Why? Well, from my point of view as reporter (as user), I’ve reported issues and after 2 years I got contacted requesting more info. We’re talking about 3-4 Gnome releases. If that bug was really important for me, either I’ll have a workaround (or even a fix and I put that on the report) or I won’t be using that software any more.
The triage process must happen as soon as possible, otherwise it’s better to close the bug.
@Michael: Starship Troopers (the first, let’s not talk about the sequels) was, in my humble opinion, great satyre/sarcastic entertainment. Below the surface of that movie is of course the allusions to imperialism, fascism, intolerance and so on. I find them doubly funny in the context where GNOME developers are regularly accused of being exactly that: fascists, nazis, freedom haters. I chose those images over gratuitous lolcats (or nothing at all) because they have meaning to me in that way.
My personal understanding of the deeper, second-second interpretation of Starship Troopers has always been exactly the opposite of “this movie promotes/endorses fascism”. So take the use of such imagery simply as tongue-in-cheek humor.
And yeah, I tend to write long blog posts sometimes to make sure I cover all bases. This is why I break them apart with pictures; do not underestimate the workings of the mind! I, for one, have a natural tendency to gloss over big walls of text with no images, and I’m surely not the only one. Thanks for your feedback though 😉
@nekohayo: Have you ever tried splitting a blog post up into a series of blog posts? The separation of text sections is usually accomplished with meaningful headlines.
You have headlines, but they are too small/don’t stand out enough. Try adding the space of 3 or 4 newlines on top of each new headline, and see how everything becomes more readable. No need for pictures to take over the job of a couple well-placed newlines and a bit of typography 101.
@nekohayo re: Starship Trooper — see how you got sidetracked by your own pictures now? You assume I want to talk about Starship Troopers all of the sudden. I don’t.
GNOME: After the desktop we’ll get to your blog posts. And pictures. Soon.
Thanks for the thought-provoking blog post, nekohayo. I’d bring it to Wikimedia’s bug wrangler’s attention, but I’m fairly sure Andre Klapper still reads the Planet! 🙂
In the case of my project, I know that better tools (using Bugzilla better, for instance) is going to help us get rid of unreproducible bugs. I wonder whether there are tools that could help get an answer to this question: How high-quality is the bug database overall? That is, what proportion of open bugs contain adequate reproducibility steps, and are currently reproducible?
A question that needs more of a leader’s time to answer or decide: what are the strategic priorities of our community and our product? That decisive priority-setting is more present in some communities than others — thanks for reminding us that indecision is quite a killer.
Michael Hasselmann: you’re being a bit unpleasant. Lighten up. 🙂
Reviewing from time to time our own bug reports and feature requests is also important: we can usually give more information, close the bug, or just say that the bug still occurs.
Several years ago I was a simple user and I filed some feature requests that I have closed recently by looking at my list of opened bugs.
Great post! I agree with your point of view, since a project with many bugs is sometimes difficult to fix or to find people able to understand and fix the bugs.
Also, I’d like to ask for your help with Gtranslator. The maintainer has nearly forgotten about it, and I’m looking for people who could help with some bugs (and also maybe becoming the new maintainer). Maybe you could help us reviewing some bugs. There are not critical bugs reported, but some of them make the tool a bit uncomfortable to use.
Please feel free to contact me by mail if you could hep us. I’m trying to save this app in benefit of all GNOME translators.
Many thanks for your help!
There are many steps and ways to clean up a database, like going through your own tickets or reproducing old tickets that have not seen changes for a while, or tickets that have not seen any comments except for the reporter her/himself. These queries are not totally trivial to create though (and the last example won’t work in GNOME Bugzilla as it’s too old).
WONTFIXing enhancement requests as a triager is harder though as you need to know the developers’ plans. In Evolution I use an “evolution[wontfix?]” whiteboard entry to propose to developers.
Nautilus and Evolution have seen great cleanup this year, see the charts here and here
@Sumana: It’s a case of “on the internet, no one sees you smile”. I am not a friend of writing fluffy criticism. Instead, I try to get my point across, with hopefully as few words as possible, and as logical as possible.
The resulting brevity is often seen as arrogant or harsh, but that’s OK. That’s still better than being misunderstood on your original criticism.
Maybe some gnome developers could make some google code-in tasks that help close bugs (assuming that they are a mentoring organization this year). It is a great oppurtunity to close expired bugs/easy bugs and maybe add to the contributers for the project.
I’ve got lots on my plate already. I’m sorry but I can’t help with gtranslator: I can’t triage the bugs because I am not familiar at all with the application (I cannot triage what I don’t know and use) and I won’t be coding on it because it’s a hundred thousand lines of C code and I’m just an occasional Python hacker.
On a sidenote… it “only” has about 75 bug reports, of which half are feature requests! About 40 of the bug/feature reports are over 500 days old. Shouldn’t be very hard for the translation team, those who actually use the app, to review them for relevancy with the 2.91 version. That being said, your problem here is not “the maintainer’s vision is blurred by a huge amount of bug reports” like the other projects I mentioned in this blog post; your problem is actually finding a maintainer.
Hi, Thanks for the great blog post.
I have been involved in a bit more than 1000 bugs in the last ten years. It rarely happens that I do not report an issue that I see.
My bug reports are spread over the bugzillas at Novell, red hat, freedesktop, Mozilla and canonicals lanchpad.
One think that is very unfortunate is the restrictive access permissions in gnome bugzilla. I cannot not close bugs, nor can I change versions, and other important information on anything but my own bugs.
This is discouraging. I have come to the gnome irc channel several times in the last months just to get reports closed. You cannot expect this from people that quickly want to duplicate some bugs they found while searching their problem on google.
I think it would be best if reports would have to be reproduced with current software versions every six or twelve months. Their version is updated in this process.