Last week, Lars Lindner provided us with an early christmas gift when he announced that a new stable release of our beloved feed reader was now available. First, I would like to applaud him for his continued efforts over the years in maintaining this handy and high quality application. Many users like me have been patiently waiting for the 1.8 release in the hope that it would finally solve the infamous performance problems of the 1.4 and 1.6 series. However, while I can’t speak of the performance once the app is launched, the startup performance has not met my expectations (in some cases, it has regressed). This blog post is intended to share my findings (which I discussed with Lars) and to invite you to comment on possible solutions.
Here’s an introduction to the problem:
- An average Liferea user is expected to have hundreds of feeds and thousands of unread items (I have about 200 feeds and 2500 unread items). From discussions I’ve had, this is not considered excessive by Liferea standards.
- Liferea uses SQLite as a database for storing feed data into a single “liferea.db” file. Mine currently weighs 115 Mo in its vacuumed form. SQLite has this terrible tendency to suck because you need to vacuum (compact) it every once in a while or the performance degrades terribly. Firefox has the same problem.
- There also was ext4 as a suspect, although from all my tests on that matter in the 1.6 series, ext4 was definitely not the culprit. An ext3 partition had the same performance problems (or, at least, the same Liferea startup times).
On a beefy machine with a Core2Quad CPU, 4 GB of RAM and SATA2 hard drives, the only thing that made my 45-50 seconds startup time go down was… buying a solid state drive. Then, the startup time went down to about 10 seconds. I could live with that.
Enthusiastic about the prospect of a 1 second startup time (and a pony), I did not wait for Liferea to be packaged in my favorite distro and compiled it myself. Then I realized that not only had startup times stayed mostly unchanged on a conventional hard drive, they actually increased with the solid state drive. Here are my general startup times (with repeated measurements to ensure validity):
|Hard drive type||Cold start||Warm start|
|HDD||38 secs||35 secs|
|SSD||21 secs||19 secs|
The hard drive times went down slightly, but the SSD times went way up (compared to a 10 seconds cold start time in 1.6). Whatsmore, for both the HDD and SSD, the “warm start” times are almost the same as the “cold start” times! What’s going on here?
Luckily, liferea has a built-in profiling system. You can use “liferea –debug-performance”, or even “liferea –debug-performance –debug-db” to get measurements of individual operations. Liferea will tell you if any particular function call is slow. I’ll spare you the details of my analysis (ie: how I determined the total count, which functions to ignore/merge together, etc. This is left as an exercise for the curious reader) and show you the pretty executive charts:
As we can see, the things that stick out first are VACUUM, doing the “default source import” (parsing the .opml xml file containing the list of feeds?) and counting the unread items. I have about 200 feeds, each requiring 0.02 second to count unread items on a conventional hard drive (notice how the solid state drive completely nullifies this? :).
…Wait a second, VACUUM?! In previous Liferea releases, vacuuming the database was never done by the application, users had to do it themselves. But now, Liferea 1.8 vacuums automatically on startup.
Let’s represent this breakdown in a different way:
Vacuuming eats between 33 and 80% of the startup time (HDD vs SSD). We now have some explanations for our initial observations:
- Liferea has seen some performance optimizations that probably improve startup times.
- Those using solid state drives do not truly benefit from these optimizations, but are heavily affected by the newly added “vacuum” operation. Result: startup times went up.
- For users with conventional hard drives, the optimizations improved startup times (I think) but vacuuming increased startup times, thus masking those improvements. Result: startup times stayed mostly the same.
By disabling the vacuum (commenting out the code), Liferea now starts in 4.48 seconds on my computer. That’s night and day.
Back in 2008, Lars himself thought that vacuuming automatically was not a good idea:
“The problem with it is that it also takes very long. With a 50MB DB file I experienced a runtime of over 1 minute. This is why this can be only a tool for experienced users that know how to do it manually knowing what to expect. For executing such a long term operation automatically on runtime would surely be unacceptable to the unsuspecting user. Also there is no good way how to decide when to do a VACUUM to save disk space and improve performance.”
Here’s a key rule about software performance optimization: performance problems are very often “needless work done at the wrong time”.
I’m no database expert (correct me if I’m wrong), but as I understand it, running vacuum everyday provides no significant benefit (it only provides a benefit when there’s a significant amount of stuff to vacuum). If you’re vacuuming on every startup, you’re using a nuclear weapon to dig a hole in a small garden.
Well, here’s the problem:
- You can’t vacuum while you’re actively using the application (as it blocks the database).
- You can’t vacuum when Liferea exits. According to Lars, “Liferea is too often killed by the session manager or [computer] shutdowns”.
- You can’t vacuum on startup “only once in a while” because, according to Lars, “if a program is sometimes slow (because of VACUUM) and sometimes not, users will complain or cancel the slow startup. So not doing it everytime is no real option.”
Short of finding a magical database backend that never hits those kinds of issues, here’s the potential solution I came up with:
- We already have a way to measure the startup times. Make it so that Liferea measures its startup time on every startup even without debugging mode.
- Factor in the count of items in the database, and the filesize of db file. [insert clever mathematical formula here]
- Once startup is done, if the startup time was unreasonably/statistically slow, offer to vacuum the database to improve performance.
An unintrusive infobar widget could be shown to the user:
Yes, imposing the responsibility of database performance optimization onto the user sounds like a bad idea… but I think we can agree that Liferea is a special case: it is an application that is meant to handle huge amounts of ever-changing data over time. It is not entirely unreasonable, in my humble opinion, to present the user with this occasional, unobtrusive prompt.
Of course, we should avoid nagging the user about it:
- If the user clicks Optimize, do not ask again for at least a few weeks/month.
- If the user clicks “Later”, do not ask for a few days.
As I said in the beginning, this blog post is intended as a summary of my findings and as a call for the collective wisdom of the hive. I know there’s a lot of very smart developers out there who might know of a solution we haven’t thought of.
What do you think? Is there a better way (be it in terms of backend scalability or in terms of UI/UX)?