Last week, Lars Lindner provided us with an early christmas gift when he announced that a new stable release of our beloved feed reader was now available. First, I would like to applaud him for his continued efforts over the years in maintaining this handy and high quality application. Many users like me have been patiently waiting for the 1.8 release in the hope that it would finally solve the infamous performance problems of the 1.4 and 1.6 series. However, while I can’t speak of the performance once the app is launched, the startup performance has not met my expectations (in some cases, it has regressed). This blog post is intended to share my findings (which I discussed with Lars) and to invite you to comment on possible solutions.
Here’s an introduction to the problem:
- An average Liferea user is expected to have hundreds of feeds and thousands of unread items (I have about 200 feeds and 2500 unread items). From discussions I’ve had, this is not considered excessive by Liferea standards.
- Liferea uses SQLite as a database for storing feed data into a single “liferea.db” file. Mine currently weighs 115 Mo in its vacuumed form. SQLite has this terrible tendency to suck because you need to vacuum (compact) it every once in a while or the performance degrades terribly. Firefox has the same problem.
- There also was ext4 as a suspect, although from all my tests on that matter in the 1.6 series, ext4 was definitely not the culprit. An ext3 partition had the same performance problems (or, at least, the same Liferea startup times).
On a beefy machine with a Core2Quad CPU, 4 GB of RAM and SATA2 hard drives, the only thing that made my 45-50 seconds startup time go down was… buying a solid state drive. Then, the startup time went down to about 10 seconds. I could live with that.
Enthusiastic about the prospect of a 1 second startup time (and a pony), I did not wait for Liferea to be packaged in my favorite distro and compiled it myself. Then I realized that not only had startup times stayed mostly unchanged on a conventional hard drive, they actually increased with the solid state drive. Here are my general startup times (with repeated measurements to ensure validity):
| Hard drive type | Cold start | Warm start |
|---|---|---|
| HDD | 38 secs | 35 secs |
| SSD | 21 secs | 19 secs |
The hard drive times went down slightly, but the SSD times went way up (compared to a 10 seconds cold start time in 1.6). Whatsmore, for both the HDD and SSD, the “warm start” times are almost the same as the “cold start” times! What’s going on here?
Luckily, liferea has a built-in profiling system. You can use “liferea –debug-performance”, or even “liferea –debug-performance –debug-db” to get measurements of individual operations. Liferea will tell you if any particular function call is slow. I’ll spare you the details of my analysis (ie: how I determined the total count, which functions to ignore/merge together, etc. This is left as an exercise for the curious reader) and show you the pretty executive charts:
As we can see, the things that stick out first are VACUUM, doing the “default source import” (parsing the .opml xml file containing the list of feeds?) and counting the unread items. I have about 200 feeds, each requiring 0.02 second to count unread items on a conventional hard drive (notice how the solid state drive completely nullifies this? :).
…Wait a second, VACUUM?! In previous Liferea releases, vacuuming the database was never done by the application, users had to do it themselves. But now, Liferea 1.8 vacuums automatically on startup.
Let’s represent this breakdown in a different way:
Vacuuming eats between 33 and 80% of the startup time (HDD vs SSD). We now have some explanations for our initial observations:
- Liferea has seen some performance optimizations that probably improve startup times.
- Those using solid state drives do not truly benefit from these optimizations, but are heavily affected by the newly added “vacuum” operation. Result: startup times went up.
- For users with conventional hard drives, the optimizations improved startup times (I think) but vacuuming increased startup times, thus masking those improvements. Result: startup times stayed mostly the same.
By disabling the vacuum (commenting out the code), Liferea now starts in 4.48 seconds on my computer. That’s night and day.
Back in 2008, Lars himself thought that vacuuming automatically was not a good idea:
“The problem with it is that it also takes very long. With a 50MB DB file I experienced a runtime of over 1 minute. This is why this can be only a tool for experienced users that know how to do it manually knowing what to expect. For executing such a long term operation automatically on runtime would surely be unacceptable to the unsuspecting user. Also there is no good way how to decide when to do a VACUUM to save disk space and improve performance.”
Here’s a key rule about software performance optimization: performance problems are very often “needless work done at the wrong time”.
I’m no database expert (correct me if I’m wrong), but as I understand it, running vacuum everyday provides no significant benefit (it only provides a benefit when there’s a significant amount of stuff to vacuum). If you’re vacuuming on every startup, you’re using a nuclear weapon to dig a hole in a small garden.
Well, here’s the problem:
- You can’t vacuum while you’re actively using the application (as it blocks the database).
- You can’t vacuum when Liferea exits. According to Lars, “Liferea is too often killed by the session manager or [computer] shutdowns”.
- You can’t vacuum on startup “only once in a while” because, according to Lars, “if a program is sometimes slow (because of VACUUM) and sometimes not, users will complain or cancel the slow startup. So not doing it everytime is no real option.”
Short of finding a magical database backend that never hits those kinds of issues, here’s the potential solution I came up with:
- We already have a way to measure the startup times. Make it so that Liferea measures its startup time on every startup even without debugging mode.
- Factor in the count of items in the database, and the filesize of db file. [insert clever mathematical formula here]
- Once startup is done, if the startup time was unreasonably/statistically slow, offer to vacuum the database to improve performance.
An unintrusive infobar widget could be shown to the user:
Yes, imposing the responsibility of database performance optimization onto the user sounds like a bad idea… but I think we can agree that Liferea is a special case: it is an application that is meant to handle huge amounts of ever-changing data over time. It is not entirely unreasonable, in my humble opinion, to present the user with this occasional, unobtrusive prompt.
Of course, we should avoid nagging the user about it:
- If the user clicks Optimize, do not ask again for at least a few weeks/month.
- If the user clicks “Later”, do not ask for a few days.
As I said in the beginning, this blog post is intended as a summary of my findings and as a call for the collective wisdom of the hive. I know there’s a lot of very smart developers out there who might know of a solution we haven’t thought of.
What do you think? Is there a better way (be it in terms of backend scalability or in terms of UI/UX)?
You can check out my main website, find me on Twitter or Mastodon or subscribe to my YouTube channel.
Comments
28 responses to “Investigating Liferea's startup performance”
I can’t help you brother.
Tried Liferea and all I get is crashes or bad parsing.
Google Reader works, no matter if it is hosted by the Devil.
I don’t think the user should have to deal with that. Especially with technical terms like “database”. Why not a cron job running when liferea is not ?
I’d probably look to see how Firefox handle it. They don’t have slow startup times anymore, and presumably also VACUUM the database or equivalent at some point.
Given the developer has made the choice to use SQLite, I don’t see why you’d store the list of feeds in XML. Database queries would be much faster than parsing XML…
I agree with Francois that asking the user about optimizing the database is a very bad idea.
@Name: sad to see you’re experiencing crashes. I don’t remember ever experiencing crashes in years of use (which is quite surprising when you think about it).
@François: really, is “database” such a technical term? I’d argue that people who use feed readers are pretty technology-savy people, or at least technology-inclined. Not your stereotypical grandma. And these days, even your average Joe on the street tends to have some rough idea of what a database is (“a place where the computer puts a bunch of data”).
@Stuart:
Last time I checked, Firefox never vacuums, ever. Performance starts to go down the drain when your places.sqlite (or some other file) becomes too big. I’d love to be proven wrong and be told that they finally fixed this recently. On the other hand, after all these years they still haven’t learned how to free my RAM either, so I’m not holding my breath 🙂
For the list of feeds in XML: I’m guessing this is probably because the xml file is a standard OPML file, which makes importing/exporting/interoperating with other aggregators much easier, and perhaps a safer choice (it might be more corruption-resistant than a binary database file? I don’t know)? I presume Lars can answer that one.
The problems I can imagine so far with the cron job approach:
– you can’t have it happening while liferea is using the db.
– users would start complaining that liferea randomly pegs their hard drive and CPU out of nowhere. And boy do we know that the desktop starts being sluggish as soon as there’s a significant amount of I/O since some point in the Linux 2.6 history.
Why not just automatically vacuum every X days when the program is idle?
Doing this when minimised shouldn’t be a problem and it is quick enough that the user shouldn’t even notice (let alone complain) about the CPU/disk usage.
If you still feel this could be intrusive you could do a vacuum when the screen saver is activated and then prompt the user if the screen saver never gets activated for some reason (such as if it is disabled)
I’ve been thinking for a while about how to emulate an online vacuum as a feature of SQLHeavy (an SQLite wrapper I wrote)… I think it should be possible to use SQLite’s online backup API to create a copy of the database, start saving all executed queries and vacuum the backup, replay all the saved queries in the backup database, and finally replace the original with the backup and switch to using it.
That said, I’ve never really been convinced that SQLite is a good fit for Liferea… if I were them I’d be taking a very close look at something like Kyoto Cabinet, Tokyo Cabinet, LevelDB, hamsterdb, etc.
I know that “database” is not such a technical term. But I still think that the average user of a feed aggregator doesn’t necessarily have to know that the software is using one. To him, it’s just a interface to read news ! A list of items. Where is the database in that ? Is Firefox ever mentioning “database” ? I don’t think so.
And more generally, I don’t think the user should have to deal with this kind of cleaning. If he has to do it, it means that the software should be improved. It’s surely not to blame the authors of Liferea, I would say the same about any other software.
If this vacuum operation is really too long on our recent computers, where this database only contains a few thousand items (it’s nothing for a database), it means that something is wrong. Maybe SQLite is not the best choice. I don’t know. But in any case, the lambda user should just use the software and not have to deal with anything like that. I stopped using Liferea because of this long startup. And now I’m using Google Reader, I’m never asked to do anything to make it work faster. No simple software should do that. No matter if the user knows about what is a database or not. Some people I know are using feed aggregators, and even if they know what is a database, they have no idea what is inside, how it’s working, and what a vaccuum operation is about.
I think that’s a great idea. I can’t think of anything better. Any kind of automatic method will inevitably clash with the random usage patterns of users, at the worst possible time too. (See Murphy’s Law.) (Ie. if you make it start vacuuming while the screensaver is activated, I guarantee you the user will be in a mad rush, a few minutes before a major assignment is due, trying to log back in to upload his work, and miss that deadline because his browser was to slow to load.)
I highly doubt a simple polite notice somewhere, like the one you mentioned, will hurt people in any significant way.
I have also noticed liferea’s pretty aggressive use of disk-storage, so I ended up moving the data (especially liferea.db and the cache/favicons) to memory first (/dev/shm here) — and it runs remarkably smoothly now :P. Of course, this is just a temporary solution. (Perhaps more things should be kept in memory for longer, before getting synced to disk?)
It’s not that hard. First of all, don’t punt to the user. Users don’t know what the heck you want anyway. All that dialog says is “Do you want the app you’re using to be unusable? [Now] [Later]” I can tell you which option I will chose all the time.
You could…
… have vacuuming that doesn’t block the app.
If you must, do a copy of the db, vacuum the copy, delete the original and use the copy instead. While this operation is in process, record all actions the user does, so you can play them back when the copy is done.
… use a DB that doesn’t require vacuuming
No idea if that exists, but it doesn’t sound like a bad idea to me to implement a DB that doesn’t suck. Heck, somebody should fix sqlite to auto-vacuum and/or provide steps to do incremental vacuuming. It sounds like aqlite has a really crappy GC.
… use a smarter database layout
Using multiple databases that you can vacuum independantly sounds like a smart concept to me. You could even store posts in separate files to reduce the DB load
… don’t use a database at all
I’m sorry, but 2500 is at least 1, probably 2 orders of magnitude before you need a database and it’s certainly a place where you can keep everything in memory. So this sounds like a clear case of “doing it wrong”.
Maybe off-topic: do your startup times change when you use eatmydata or otherwise disable fsync/fdatasync?
As for ext3, since latest Fedoras use ext4 driver for all ext* filesystems, they (ext* filesystems, ed.) have similarily bad performance with applications heavily using fsync/fdatasync.
@Benjamin Otte: SQlite supports autovacuum for quite some time now (since release 3.1).
@nekohayo: Firefox does vacuum places.sqlite since some time (i think at least since Firefox 4). There is a key called storage.vacuum.last.places.sqlite in about:config which stores the unix time stamp of the last vacuum (about five days ago on my machine) and manually vacuum’ing places.sqlite doesn’t free much space (it did on Firefox 3).
I don’t know how exactly they determine when to vacuum, but i guess they use some simple time-based scheme (e.g. vacuum every week). Some ideas are here https://wiki.mozilla.org/Firefox/Projects/Places_Vacuum and in the bugs linked there.
@Benjamin (and François):
I don’t know if you use liferea (Benjamin), but my guess is that it stores/caches the entirety of the contents of each feed item into the db (I may be wrong on that assumption, but otherwise I have a hard time imagining why my database would be ~115 MB vacuumed).
I’m suspecting it’s not a mere few thousand database entries, it’s the few thousand database entries *with full text contents associated with them*. Certainly you don’t want to redownload everything on every feed refresh/startup, and one of the key features of a desktop feed reader is that it works even when you’re offline.
@Bronte: thanks for the update 🙂
@Oleg:
Well, back when I did my test with comparing liferea ext4 vs ext3, it was a year ago and I was running Ubuntu at that time.
http://sqlite.org/pragma.html#pragma_auto_vacuum does not really convince/strike me as significantly better than a manual vacuum performance-wise, but I’ll let you judge.
See also http://sqlite.org/lang_vacuum.html and http://liferea.blogspot.com/2008/08/why-auto-vacuum-is-no-good.html (though I can’t say for sure if Lars meant sqlite’s autovacuum feature or the “notion of liferea automatically vacuuming” in that blog post).
What you can do is to measure database fragmentation. If the fragmentation is low, no need to do VACUUM-ing.
Pseudo code:
1. get total pages count with “PRAGMA page_count”
2. get free pages count with “PRAGMA freelist_count”
3. calc fragmentation percentage as 100*(float)nFreePagesCount/nTotalPagesCount
4. if percent is higher than some hardcoded limit (15% for example), VACUUM the database.
Opera Mail does a similar trick quite nicely, though at app exit rather than at startup. What it does is pop up a dialog saying “Opera Mail wants to perform some maintenance. This can take as long as xx minutes. Would you like to let it perform the maintenance now?” [Yes| No]
A similar thing for the feed reader doesn’t appear unreasonable: you determine whether or not database maintenance should *probably* be performed, if you need to do it you pop up a dialog asking the user for permission so the user knows why it will take some time to load.
If the vacuuming can be canceled, all the better: the app could start vacuuming and cancel it if the user doesn’t consent.
The user shouldn’t be involved because there isn’t much
input he can give.
You already knows roughly when the cleaning needs to be done:
when the application has been used for a while. When it’s time,
just do it. (Bonus points for not bothering the rare user every
time he start the application! Pick a minimum number of startups
as well as a minimum number of days.)
Just show a message like \I need to do some housekeeping,
but don’t worry it’ll only take a minute or two.\ and start right
away. If you want to get fancy (and if the process can be aborted
halfway through) add a \Do it next time instead\ button.
If your app does housekeeping when I can see it, it is broken. Examples:
– yum
Anybody here not annoyed by yum refreshing the database index?
– Windows Update
No, when I shut down my computer, you should be shutting down and not tell me that you’re installing updates.
– GiMP
Every time I start it, it’s looking for data files. Why doesn’t it just start?
– NetworkManager
No, it’s not worth waiting for IPv6, there is none. Just like last time.
I know that it’s an art to not annoy users with things that take a while. But it’s always worth it to try very hard. And punting the decision to the user is better, but still shows that the developer is not smart enough to fix the problem himself:
http://www.joelonsoftware.com/uibook/chapters/fog0000000059.html (The part about the Windows Help dialog)
How about counting unread messages in the background? Would it be possible to start reading the first feed before the others have been fully updated?
I fixed the startup time by removing the VACUUM.
Patch can be found here:
http://www.vakuumverpackt.de/reducedstartuptimeforliferea/
I have Ubuntu One syncing the .liferea config folder (1.6) and I also have an indicator applet monitoring the sync process for Ubuntu One… once Liferea starts, within a few seconds, the applet is reporting over 1000 files which need syncing.. how can this be?… it would certainly explain the performance issues and hangs on startup…
Copy the database. Vacuum the copy in the background. Merge the differences once the vacuum is done.
Had the same idea, as anon. I cant only +1 to him:)
I would at least check the following things too:
Is there enough free space on the given partition? (free space = current database file *2)
So you can do the job when there is little user interaction, in the background.
Informing the user is still a good idea. I hate when programs imagine himself smarter then myself. (not because Im smart, just maybe I know what I want).
An infoline could be displayed:
The program runs some housekeeping in the background, you can accelerate it by clicking [HERE]. (this message will hide in 5 sec).
Best,
Laszlo
Just another 2 cents:
gmail does a really great job informing the user, especially undoing.
(btw I hate the force upgrading to their newest crapiest interface, but still notification is a top-notch).
Vacuum when Liferea is running and idle. If user interacts with Liferea, pause vacuum. If Sqlite doesn’t support pause/resume vacuum, restart the vacuum when idle again. Vacuum a database copy and merge if required.
That kind of time spent on modern hardware sounds completely nuts. It sounds like an awful lot of data – are you sure it’s all legit? If so, if sqlite can’t handle that amount of data, move some of it out and stick it elsewhere (maybe in a BerkeleyDB clone).
If you have time to fix it, thinking about punting this kind of decision to the user is just silly. Really, your storage solution can’t handle 100-200 MB in less than 10 seconds? Really? REALLY?
IMHO storing full data together with metadata is a design flaw. sqlite should only handle metadata, the filesystem is good enough for storing big chunks of data.
What I think is better:
* Create a cache directory in ~/.cache/liferea (or ~/.config/liferea if it must *never* be discarder).
* For each post read, create a unique file in that dir (mkstemp()) and store full data there.
* Add the unique filename to the post’s entry in the sqlite db.
* Never VACUUM on startup (startup must be instantaneous).
* Almost never readdir() that dir or fstat() all files, a simple fopen() should be enough and it’s blazing fast even on million-files directories (I’ve tested it on extremely large maildirs).
I don’t see any improvement from v1.6. Startup takes forever, sometimes Liferea even crashes when I refresh the feeds, and I must reload all favicons before they’re displayed properly. Any suggestion? I’m on Ubuntu 11.04 HP Mini notebook. Thanks.
I’m still using Liferea 1.6 on ext4 and SSD. I found out I could reduce startup time significantly with “eatmydata” (disabling fsync for Liferea).
I’ve got a 53M liferea.db and before, I measured launch time to 3.8 sec, with eatmydata it launches in 1.3 sec. (Both timings with warm caches).
Anyway, thanks for investigating this—It’s needed to inspire change.