performance – The Open Sourcerer

2020-10-31

L'indicible horreur des fichiers SQLite (et les mandelbugs de corruption de profil Firefox)

En cet Halloween, je vous partage aujourd’hui une petite histoire d’horreur et quelques astuces, fruit de mes observations sur un mystérieux phénomène survenu en 2017. En fait, ce billet est resté dans mes brouillons depuis trois ans et trois semaines (essayez de battre ce record, bande de flemmards), et je me suis dit qu’il vallait mieux que j’en finisse avant la réélection de Trump, et publie ce billet enfin… et puis hein, ho, ça fait une éternité que j’ai bloggé en Français et sur Planet Libre, alors coucou Planet Libre, ça fait un bail. Regardez, j’ai ressuscité GTG cette année, c’est bien, ça fait le café. D’ailleurs, gardez l’oeil ouvert sur mon blog car il y a du nouveau qui s’en vient, même si c’est pas forcément en Français (et donc pas sur Planet Libre)…

2014-01-04

Scratching some Media Library itches

Besides catching a cold and shovelling snow, this holiday season I spent some time scratching itches in Pitivi. For starters, thumbnails generation: if you’ve been using the new Pitivi, you certainly ran into this:
medialibrary - partial thumbs
As you can see, thumbnails were not necessarily being shown for all clips. That was not the result of a buggy algorithm or anything like that (I don’t code often, but when I do, it’s perfect, of course), simply the fact that my solution for bug 432661 was partial: it would fetch the thumbnails if they already exist, but not create new ones.
Well, this is solved now. No more need to browse the folder with Nautilus to generate thumbnails! This does not block the user interface, thanks to Alexandru Băluț who helped me finish the job by correcting my incorrect attempt at threading with his own implementation.
medialibrary - complete thumbs
The only downside of the threaded version is that you lose the simple benchmarking system I had put in place, but no matter: I’ve measured the numbers already, so let me share them with you.
As you know, I’m pretty adamant about maintaining good performance and responsiveness in Pitivi, especially for operations such as importing clips or loading projects. I know that iconview can be slow, so my commit (before Alex’s changes) had various timers to check if searching through the iconview or updating its items was a significant performance problem. I tried with two scenarios: importing 68 clips to thumbnail, and importing 485 clips to thumbnail. Here are the resulting processing times in seconds:

	68 thumbnails	485 thumbnails
Generation	20.131	141.059
Model search	00.1186	004.8595
Model insertion	00.0432	000.4

Continue reading “Scratching some Media Library itches”

2012-12-18

GNOME 3 and login performance

How about we revive the Performance wiki page and make it a goal for GNOME 3.8 (or 3.10) to finally reach our 2005-2007 target of a “3 seconds login time”?
Our current login performance is pretty bad. We do way too much I/O and processing. If you write an application or service that automatically starts at login, please take a long hard look at how much extra work you’re doing on a cold start. It might seem small, but it all adds up very quickly with the rest of the applications competing for resources, as you can see in the bootcharts I made for that bug report:

This is not a new problem. It’s been there since the GNOME 2.x days and has not vanished (as I hoped it would) with the GNOME 3 session. It’s something that we have to tackle and solve together, as a well-integrated desktop environment and application ecosystem.
See for yourself: log-in on a 3-5 years old computer or netbook/nettop (not on a Core i7 with a solid state drive!), and you’ll feel the pain. And then, at the other end of the spectrum (a modern computer with a SSD), we should probably have the goal that login takes 0.5 seconds or less.

2011-12-24

Investigating Liferea's startup performance

Last week, Lars Lindner provided us with an early christmas gift when he announced that a new stable release of our beloved feed reader was now available. First, I would like to applaud him for his continued efforts over the years in maintaining this handy and high quality application. Many users like me have been patiently waiting for the 1.8 release in the hope that it would finally solve the infamous performance problems of the 1.4 and 1.6 series. However, while I can’t speak of the performance once the app is launched, the startup performance has not met my expectations (in some cases, it has regressed). This blog post is intended to share my findings (which I discussed with Lars) and to invite you to comment on possible solutions.
Here’s an introduction to the problem:

An average Liferea user is expected to have hundreds of feeds and thousands of unread items (I have about 200 feeds and 2500 unread items). From discussions I’ve had, this is not considered excessive by Liferea standards.
Liferea uses SQLite as a database for storing feed data into a single “liferea.db” file. Mine currently weighs 115 Mo in its vacuumed form. SQLite has this terrible tendency to suck because you need to vacuum (compact) it every once in a while or the performance degrades terribly. Firefox has the same problem.
There also was ext4 as a suspect, although from all my tests on that matter in the 1.6 series, ext4 was definitely not the culprit. An ext3 partition had the same performance problems (or, at least, the same Liferea startup times).

Continue reading “Investigating Liferea's startup performance”

2011-09-14

Pitivi's startup time

So, since it seems everybody’s been talking about startup time these days, I’ll admit I tend to secretly obsess over that too. Well, it’s not so secret given that I’ve blogged about profiling work on Specto and PiTiVi before… anyway.
I believe you should provide your developers with the fastest computers to do the development, and five years old computers to do the testing. I therefore do my benchmarks on three computers:

Kusanagi, my most powerful computer, equipped with a solid state drive and a Core2 Quad processor, 4 GB of RAM
Kuze, a six years old Thinkpad T43p, with a conventional IDE hard drive and a Pentium M processor (single core), 2 GB of RAM
Krimson, a modern laptop from 2009, with a conventional SATA hard drive and a “Pentium” SU4100 (dual core) processor, 3 GB of RAM

Continue reading “Pitivi's startup time”

2011-04-24

Negotiating Performance

If you do more than basic video editing, you may have experienced those infamous “not-negotiated” errors, one of the most annoying current issues in pitivi. Users run into not-negotiated errors when there are transitions, gaps in the timeline, and effects that require compositing (such as the “alpha” filter) fail to work properly. Sometimes, if you keep changing the timeline, things magically fix themselves (when it goes back to using whatever format it was using before it errored out).

This blog post is an attempt to explain what’s going on, based on a discussion that occurred on the #pitivi IRC channel last month. I believe it can be quite interesting to the more technically-inclined among you, and perhaps you might have ideas that we haven’t thought of so far.
As an introduction, I shall quote Alessandro:

“The problem is that the fix is not obvious. Ages ago, our pipelines where all ARGB. Then we added an optimizations to skip the -> ARGB conversion when possible, and it does make performance much better in some common cases, but it introduced those errors since videomixer can’t renegotiate format at runtime — which we try to do.”

Continue reading “Negotiating Performance”

2011-04-07

GNOME 3.0's RAM usage

…is surprisingly low. Unlike what some people would make you believe, GNOME Shell & friends don’t eat 883 MB of RAM. As you can see below, baseline memory usage is under 120 MB… And you know what? That’s less than the amount of memory that GNOME 2.30 uses on startup on Ubuntu 10.04 LTS (127 MB+ even if you cut down on some useless services).

My five-years-old “frankensteined” machine (512 MB of RAM, broken keyboard, broken CPU fan, broken touchpad) runs GNOME Shell perfectly fine for typical daily usage*, thank you:

*: of course I won’t run virtual machines and HD video editing and tons of concurrent apps on 512 MB of RAM, but that’s to be expected.

2010-11-24

Improving project loading performance

When you load a project in PiTiVi, importing the clips into the “media library” (also known internally as the “source list”) is pretty fast, but inserting them in the timeline is painfully slow. So I whipped out my torture test project and spent some time profiling what’s going on using Python’s cProfile module (I talked about it before, but see also this well-written page from the GTG folks).
The findings were interesting. Graphically, we end up with something like this:

Looking at where the time was spent, near the bottom of that graph, I suspected that linking was the root of the evil we saw.
Not long after I posted my findings, Edward posted an initial patch that makes it three times faster. Here are some measurements:

	Unpatched	Patched
“Boston summit” Project	7.9 seconds	2.5 seconds
“Texture” (torture) Project	1 min 32.4 seconds	31.4 seconds

Continue reading “Improving project loading performance”

2010-09-19

Free my memory

My fellow Antistress argued that I should be going back from Chromium to Firefox today. I replied, “Only when it starts up in less than 3 seconds and frees my goddamn memory when closing tabs”. He then replied with the following statement:

firefox a la meilleure gestion de la mémoire tous navigateurs confondus (cf tests unanimes sur la toile)

Continue reading “Free my memory”

2010-03-04

Améliorer les performances de GTG

Pour les fans de la technique GTD de David Allen, Getting Things GNOME! est une révélation, un logiciel qui rend «sans douleur» l’ajout et la gestion de tâches. Je considère les fonctionnalités suivantes comme étant celles qui démarquent GTG des autres applications:

La capacité d’utiliser un langage naturel, tel que “defer:20100224” ou “due:vendredi” (et on peut quand même utiliser les mots clés en anglais, c’est généralement plus court: “due:friday”);
La capacité de «reporter» une tâche;
Un mode «vue de travail», qui fait en sorte de masquer toutes les tâches reportées, qui ne peuvent pas encore être commencées ou qui dépendent d’autres tâches. Dans mon cas, c’est la différence entre avoir 130 tâches devant les yeux et en avoir 30. C’est fantastique, ce que ça permet et termes de «focus»;
Un système de «tags», avec la possibilité d’exclure certains tags de la «vue de travail»;
Un bouton «Marquer comme fait» qui est, naturellement, jouissif.

Continue reading “Améliorer les performances de GTG”