Negotiating Performance4 min read

If you do more than basic video editing, you may have experienced those infamous “not-negotiated” errors, one of the most annoying current issues in pitivi. Users run into not-negotiated errors when there are transitions, gaps in the timeline, and effects that require compositing (such as the “alpha” filter) fail to work properly. Sometimes, if you keep changing the timeline, things magically fix themselves (when it goes back to using whatever format it was using before it errored out).

This blog post is an attempt to explain what’s going on, based on a discussion that occurred on the #pitivi IRC channel last month. I believe it can be quite interesting to the more technically-inclined among you, and perhaps you might have ideas that we haven’t thought of so far.
As an introduction, I shall quote Alessandro:

“The problem is that the fix is not obvious. Ages ago, our pipelines where all ARGB. Then we added an optimizations to skip the -> ARGB conversion when possible, and it does make performance much better in some common cases, but it introduced those errors since videomixer can’t renegotiate format at runtime — which we try to do.”

Let’s put this into a chronological perspective:

1. Before we had transitions and compositing, performance was not an issue.
2. We added compositing (for transitions and some special effects). Because it’s done in software instead of being hardware-accelerated, this caused pitivi’s performance to drop significantly.
3. To try to minimize the impact on playback performance and avoid the wrath of users, some optimizations were added. It seems that the infamous not-negotiated errors are the side-effect of those optimizations.

It has been suggested that maybe we should revert the optimization for the time being, test, release, then put the optimization back into the development version. As you can see in the chat log below, the problem is slightly more complex than it looks:

• dmj726: so videomixer is always YUVA, and before you always did an RGB or RGBA to YUVA conversion if the source wasn’t always that?
• twi_: roughly, yes
• dmj726: so what did you do to not need the conversion, make videomixer able to handle both RGBA and YUVA?
• twi_: we force whatever comes before videomixer to have an alpha channel if needed; else we let those elements passthrough without conversion
• dmj726: so you don’t do RGB to YUV conversions at all anymore?
• twi_: not exactly. But the point is, if none of the streams have alpha < 1 (and so there’s no mixing to do), we don’t force the streams to have an alpha channel
• dmj726: and the lack of alpha channels broke it?
• twi_: changing the requirement between alpha and !alpha broke it
• dmj726: but why is this causing an issue with video mixer, does video mixer run when you don’t need to do mixing? wait, it does, based on the code. the change was in smartVideoMixer I’m guessing?
• twi_: yes. videomixer can’t change format at runtime, so if you start with no alpha and then change to alpha => not-negotiated
• dmj726: was it a huge difference in performance?
• twi_: yes
• dmj726: If it wasn’t something earthshattering, I’d say take it out until you either have a fix in smartvideomixer or can get videomixer to negotiate at runtime?
• twi_: i’d say let’s take it out anyway, since there’s little point in having something fast that can’t be released, and this thing is holding back effects from being in a release
• thiblahute: but videomixer2 should fix that or not?
• twi_: no, same limitation
• dmj726: sounds like we need a videomixer3 😛
• twi_: what we should do is to replace videomixer in the pipeline when the “alpha status” changes
• dmj726: why not just do that?
• twi_: because it’s easier said than done and someone has to do it (and have the time to do it)
• dmj726: hmm…smartVideoMixer would be able to know when a clip has been added or removed, right?
• twi_: smartvideomixer already has the logic to know whether alpha is needed or not
• dmj726: yeah, I think I just found it. Then why not have that trigger a replace of videomixer? Is it difficult to initialize a videomixer? If not, I think I see where the initialization should be called.
• twi_: replacing an element while the pipeline is playing is not that easy. It’s possible, but not easy
• dmj726: honestly, fixing this would fix a lot of pitivi’s issues. What makes it hard?
• twi_: the fact that renegotiation in gstreamer isn’t as easy as it should be, which is something that is being fixed in gst