Vertical track management rumination

Suggestion
1

This post is a rumination on the ancient and contentious topic of track management, particularly vertical re-arrangement or lack thereof prompted when I wanted to do just that. See Rearrage tracks / move tracks up/down and likely others. I am not advocating a change in feature development priorities here. I have scrolled the Suggestions category.

After reviewing many prior posts on this topic I suggest there are several competing issues, multiple use cases with very different development effort, that prior posters may not be aware or have considered. In my past lives at times I was a software developer so I am making educated guesses regarding development effort and implementation design in the following note without having examined the Shotcut source code in detail. I use the term “mute” applied to both audio and video tracks to mean remove that track from the rendering process.

Conceptual Item)

There is a fundamental difference in the rendering process between video tracks and audio tracks - audio tracks additively mix together (subject to mixer settings), video tracks overlay in a precedence order (Z-order rendered result subject to opacity parameters). In the Shotcut Timeline GUI, the relative vertical positions of video tracks is used to express the video pixel overlay precedence for non-null, non-transparent pixels. In these senses for rendering, including preview rendering, the video rendering cares about vertical video track position in the GUI whereas the audio rendering does not.

So, changing the relative vertical position of a video track is not just fiddling some GUI code - it also changes the ENTIRE VIDEO RENDERING data structure chain logic constructed in program computer memory, including the if/when logic of execution of all the dozens :wink: of associated video filters you have attached to all your clips. NON-TRIVIAL to design and program that capability so the interactive GUI does not appear to freeze while Shotcut is re-arranging the background furniture.

Video rendering)

Now clearly some of that render chain re-arrangement capability already exists in that you can move clips vertically between video tracks. However, once a track change action is completed, for smooth interactive preview GUI performance you want to pre-compute as much of the preview render data structure in memory as possible ahead of play time. With an interactive interface, typically there are an abundance of milliseconds to do this while the user is contemplating (sucking thumb or twirling the mouse cursor in circles as seen in soooooooo many YouTube vids) what to do next, especially when exploring the creative space. Then again, in a repeative, fast-pace professional edit workflow the pro user may launch preview before preview pre-comp is finished. Oops. The preview stutters as computation struggles to keep up. This is why pro workflow situations use proprietary $$$$ software on gaming monster workstations and likely why pro-experience complaining Shotcut users forget what they were using when donation-ware Shotcut does not offer the same pro features on a mid-range laptop situation.

For interactive preview performance, render computation should be “track high” because knowing the opacity for all (frame number, pixel number) tuples in a higher precedence track potentially means vastly less computation needed on all lower precedence tracks for their equivalent track tuple. An obscured pixel is one you don’t have to compute (except for possibly filter states). A clip change action means only those frames and pixels affected down the track precedence stack need be re-computed. Changing the precedence of a track means that some or all of the entire track stack pre-comp has to be re-computed. Suddenly, the interactive GUI is at risk of a freeze from that amount of pre-comp computation. There are well known programming techniques for asynchronus background computation but those necessarily come with elevated programing error risks as well.

Multiple use cases)

There is a finite amount of computer monitor vertical space budget for the Timeline GUI. If you have a lot of video tracks because of necessity or because you are exploring the creative space, then the productivity issue is you can end up constantly having to vertically scroll the Timeline pane. The inexperienced, perhaps not understanding that the rendering precedence will change, will want to vertically re-arrange the tracks to visually group the tracks they are currently fiddling, their current working set, so as to avoid Timeline pane scrolling. But also, when exploring the creative space, you can accumulate many experimental tracks you want to retain for lessons learned or fallback. Handily, you can lock those now dormant tracks so you don’t modify them by accident (soooooo easy in a drag-drop GUI) and mute them so they no longer participate in rendering. Un-handily, those dormant tracks still consume vertical GUI space. So what are the use cases?

Productivity - I just want to avoid endless vertical scrolling of the Timeline pane to see the subset of tracks I am working at the moment. Whether a given track not in my current working set is also participating in rendering is an independent track muting question.

Retaining capital investment (time) in experimental tracks - I want to be able to put away from and retrieve a track back to the Timeline GUI as a track behaviors object, not a rendered clip. When put away, a track is locked, muted and hidden - it no longer consumes Timeline GUI space.

Productivity - I’ve changed my mind on content rendering precedence or goofed and started adding higher precedence content first rather than last. This raises the point that Shotcut users have to mentally bin clips and other content by rendering precedence PRIOR creating tracks because you can’t change track precedence. The frequent user will eventually figure this out and internalize it. The occasional user will have to re-learn it every time.

The DAW I use has some handy track management features that are applicable here. One of those features is a seperate tab-pane that provides per-track for A) “Hide” a track, meaning the track substantially disappears from the DAW equivalent of the Timeline GUI, meaning you gain back that track’s vertical GUI display budget, but that track still participates in rendering subject to normal track mute controls. In a Shotcut context, the hidden track would still occupy a precedence slot in the track stack and the semantics of whether a hidden track is also by default locked against mulit-track splits and such would be TBD. B) “Disable” a track is an alias for lock, mute and hide. In a Shotcut context, a disabled track still occupies a precedence slot in the track stack.

A counter to the vertical space complaint is that the major GUI panes are detachable, so the Timeline pane could be moved to a dedicated monitor, but that only helps if you have one. Also, the buttons for detaching panes are almost un-noticable. You have to know what you are looking for and visually hunt for them. The View->Layout menu might usefully have a function to flash those buttons on the interface so they can be located.

The above use cases are then my first suggestions for intermediate steps to vertical track re-arrangement capability. They take some complaining user cases off the table before you need to tackle the performance issues from changing track rendering precedence.

Another possible intermediate step is my guess that null-content tracks likely incur negligable computation when created or when changing their video rendering precedence. They are rendering NOPs until they have content. Changing the precedence (vertical position in the GUI) of a null-content track necessarily changes the absolute precedence of all other tracks but not the RELATIVE precedence of all other tracks. Taking a cue from spreadsheet software, I imagine new track operations of INSERT ABOVE and INSERT BELOW. Now users can at least re-arrange clips in desired precedence by moving them between to newly inserted null-content tracks. Tediously clip by clip or Select All then move in bulk is better than the nothing of the current version.

Regards.

2

You do not need to use the buttons. Most people drag the panel’s non-distinct title bar. You just need to be careful when dropping to make it float, or simply double-click the panel title bar.

As for the rest of what you wrote, I think I will need to read it a few more times to understand your suggestions because they seem to be buried within many words, or I need to “read between the lines.”

I think you are suggesting the usage of additional tracks waiting in the background until needed or used for scratch space. Do keep in mind that number of tracks increases the amount of caching and memory usage because we want to avoid reopening files and decoders for every frame during a time period. Theoretically (and some people have) use up to three files per track: a transition between two videos that uses a custom file. And repeating that on every video track at the same time while also blending! I call this “avoid thrashing” and the major performance cost of that.

3

Point on dragging individual panes. However, my gentle counter-point regards that would be your own words “non-distinct title bar”. Kind of like saying we’ve visually hidden Easter eggs in the GUI that do useful things. You, occasional user, just have to guess they exist, where visually in the GUI they might be and what they might do. I suggest re-thinking that GUI design detail to improve the app learning curve. Permanently visible title bars or mouse-over title bars could be a View->Layout or Settings->User Interface option choice for the newbie or visually impaired if you don’t want distinct title bars as the default. BTW, Settings->User Interface currently will not select in the menu droplist.

Regards all the words, given the history of contentiousness on the track management topic, I thought it important to develop not only ideas and but also supporting rationals for possible intermediate steps based on my educated guess that changing the video rendering precedence of an occupied track has hard problem risks to GUI responsiveness (or maybe the issue is accumulated techinical debt?). So I explored the question of are there some track management use cases that could be taken off the table with lesser effort. That could leverage functionality already present. Many of the rational words are for the community as well as the developers. Better than just flame demanding it as some have done in the past, yes?

Regards.

4

“I think you are suggesting the usage of additional tracks waiting in the background until needed or used for scratch space. Do keep in mind…”

Chewing over your interpretation in the above paragraph, some clarification. I am not suggesting “…additional tracks waiting in the background until needed…” (1) My “Hide” mechanism is all about reducing GUI vertical real estate consumption per-track and only that, i.e., from a full height track to a bare minimum button row. The user is hiding an existing track object, not adding a new one, so, other that GUI pixels, no change in those memory resources and open file descriptors you mention or in rendering burden. SC already has this capability as Track Height->Make Tracks Shorter but in v21.06.29 that action gets applied to ALL tracks equally. The user can’t apply Track Height adjustment (and lock track height against later global track height actions) track by track. In some tracks I want thumbnails but other tracks are stable enough I don’t need that GUI real estate consumption. (2) My “Disable” mechanism could also be imagined as Close Track Object and Open Track Object. By closing a track, the user is explicitly saying I want to keep this track as a track object (the semantics of a playlist object are different) in my project but I no longer need it to consume significant interactive resources (no presence in the Timeline, appear only as a list item in a track management pane) and certainly not any decoding or rendering resources, and that if I open a closed track I understand that interactive resources must be re-aquired which may take some progress bar time. Given the troublesome issue of what should be the Z-order of opening a track we could say the semantics are that of the Track Operations->Add Video Track. (3) Separately, I also note that SC already has Track Operations->Insert Track which inserts a null-content track above the currently selected track. I suggest re-labeling this in any case as Insert Track Above, then adding the symmetric operation of Insert Track Below. (4) Lastly, my suggestions are for intermediate steps to reduce pressure for full Z-order modding capability but they still may be significant detours in the design roadmap and I’m good with that as a No Can Do reason.

Regards.

5

Not our design; it is the library’s default. Tweaks like this is not always what everyone wants or our priority. I typically defer to the defaults here since I know the library authors have invested more effort and factored in more feedback than we get. It is a very popular and mature GUI library called Qt. Maybe you have heard of it.

Settings->User Interface currently will not select in the menu droplist.

That is by design; it is a category label. All menu items under it pertain the user interface. This avoids the need to have many sub menus, which can be annoying like a labyrinth.

6

Yes, apps still whankering about adapting to Qt 5 LTS from Qt 4 and now Qt 6 is out. No, do not want to fiddle in the library.

For Settings->User Interface, ahhhhhhhh its a droplist vertical section title, as are Project and Player. Now doubt Qt again. The usability problem is no visible distinction in font or exdentation or a section marker line or something other than not selecting that signals this is a section label and not a menu item that is broken. Sigh…

Regards.