When writing a filter ui.qml file, is there a way to retrieve the dimensions of the original unscaled clip that the filter is attached to? I tried finding QML for the clip properties tab to see if I could copy what it did, but couldn’t find any related QML. I tried producer.width and producer.meta.media.width and a number of other combinations and still got nothing. Any help is appreciated.
producer.get(‘meta.media.width’) and producer.get(‘meta.media.height’)
1 Like
This information made my day. Thanks!
I celebrated too soon. producer.get() throws an error… method not found.
I’m in shotcut/qml/filters/crop/ui.qml in the Component.onCompleted method inside the If filter.isNew block.
The goal is some form of: filter.set(‘producer_width’, producer.width)
Best I can tell from Shotcut source, “producer” in this context would be a QmlProducer, which does not appear to expose the underlying Mlt::Producer in a way I can get to it. Nor does QmlProducer appear to expose the metadata directly. Am I doing something totally wrong?
To elaborate on what I’m trying to do…
The Crop: Source filter stores the crop amount in pixels. If the clip (producer) is swapped from proxy resolution to full resolution, the pixel values are incorrect (too small) for the new clip dimensions. The pixel values aren’t scaling along with the clip dimensions.
I have gotten around this by embedding the clip’s width and height in filter storage (currently hard-coding it to test it). When the filter is loaded and sees current dimensions different than the cached dimensions (meaning a swap from proxy to full resolution happened), it creates a percentage scale based on the old dimensions to calculate and rewrite the pixel values for the new dimensions. Preview and export processing continue to work as-is using existing pixel logic.
It appears that any filters whose coordinate system is based on clip dimensions will need access to those dimensions programmatically in order to create any kind of final proxy workflow solution. I’m currently not sure how to get those dimensions. Everything else is ready for the crop filter. I dived into the text filter too and so far I have everything I need from the profile object’s properties.
Don’t try to fix this in the UI code.
Fair enough. Could QmlProducer still be modified to expose producer.width and producer.height? The crop filter sets the slider bar maximum values to profile.width instead of producer.width for lack of a better alternative. On a 1080p timeline, profile.width at 1080 is less than a quarter of the width of average smartphone images let alone high-resolution cameras. The slider maximums still need to reflect the true producer width and height to have full crop ability. If you wish, I can submit the change to qmlproducer.h/.cpp as a pull request. Thanks for your time.
Sorry to be short. I was not in a position to test things and reply with more info.
I was obviously mistaken about producer.get. I was thinking about MLT class inheritance at the moment, but these are QML wrappers to these MLT classes with a different interface. These MLT properties meta.media.width and .height are on the MLT producer and frame objects but not through the QmlProducer or filter. Yes, we can add a get method to QmlProducer.
I am trying to figure out a way within MLT to serialize mlt_rect as a percentage automatically based on profile dimensions to solve this problem for most filters. The Crop: Source filter is special because it does not use mlt_rect, and its pixel units are in terms of source pixels instead of profile. Nearly everything else that is coordinate or size related and not in relative/percentage values is in profile pixel units.
There may be many pitfalls to a proxy workflow that uses smaller resolutions, and I’m just not sure it’s worth it yet or if effort is better spent on performance improvement. Most people don’t want the overhead of generating proxy and optimized media. After all, the GPU Effects mode is still there waiting for more debugging while there’s all this new work to add a proxy feature and make everything relative, with some of it not obvious how to fix (e.g. avfilters and webvfx). I hesitate to put out something else half-baked and with many caveats.
I definitely understand your concerns. I had them too. In this post, I provide a complete proxy solution with working code attached for your critique and to see how many concerns still remain using this method. Here is the hope at the end of the tunnel:
- I tried the serialize mlt_rect method and it wasn’t a desirable path
- I think I found a better way that requires little to no effort on your part
- The pitfalls have been negligible based on my 2+ years of hacked proxy experience with Shotcut
I’ve thought about proxies quite a bit and made several prototypes. I’ve attached my latest crop filter ui.qml which demonstrates the technique I’m using. Search the comments for the string “Austin” to see which parts I modified. There is minimal change to existing code. Obviously, it uses constants for the producer dimensions because producer.get methods don’t exist yet. I sent Crop to you because it’s easier code to analyze than Text or S&P, but the concepts are the same whether translating the producer or the profile.
These are my observations on proxy workflow so far (subject to further enlightenment):
Goals:
- No modifications to MLT or filter libraries (frei0r, etc)
- MLT XML backward compatibility with old projects
- Adapt to changes in profile or producer dimensions
- No loss of “coordinate system resolution” while in proxy mode
- Don’t trip the “project modified” flag unless translation actually happens
- Preferably a relaxed phase-in deployment rather than an all-consuming big bang
The “Serialize mlt_rect” Method:
This is the road I started on. I quickly discovered that managing two sets of coordinates (pixel and percentage) became a maintenance nightmare (specific reasons at the end of this post) and had potential to fail Goal #4. So I moved on to a new method.
Proposed Solution:
- Filters would cache profile and/or producer dimensions in their MLT XML storage.
- A filter’s Component.onCompleted method compares the current profile and/or producer dimensions to the cached values. If there is a difference, it assumes a swap happened and rewrites the pixel values. Here is the mental breakthrough: MLT XML and filter UI must allow pixel coordinates to have two decimal places. This is what prevents a loss of resolution when the coordinates are converted between proxy scale and full scale. Two decimal places can support full resolutions up to 16K and proxy resolutions down to 480x270 without conversion loss. Since we maintain resolution with decimal pixel coordinates, we no longer need separate percentage values. They become redundant because we can calculate the percentages from the cached dimensions. Plus, decimal pixels allow a discerning user to target “in-between” pixels when specifying a 640x360 proxy coordinate that is later scaled up to 4K.
- The filter continues to operate in existing pixel mode until an onProfileChanged event fires. Pixel values are rewritten the same way and the UI refreshed at each event fired.
- The filter UI code is written such that if a profile or producer never changes dimensions, then the “proxy math code branches” are never even executed. This maintains exact backward compatibility, makes the filter easier to maintain and debug, eliminates any rounding errors, and requires the least modification to existing filters. Users would never see decimal pixels in this scenario and would be totally unaware that proxy plumbing even existed.
Pitfalls:
- WebVfx would not benefit unless the template used its own similar mechanism. We get around this by applying those filters in full resolution mode and ignoring what they look like in proxy mode.
Bonuses:
So long as producer.width and .height methods exist, and pixel coordinates with decimals are accepted by filter libraries (or at least converted to integer by truncation rather than rounding), then no additional changes need to be made to Shotcut, MLT, or upstream filter library code. Mathematically, it doesn’t matter which layer does the coordinate translation. I’m proposing to put all translation in the UI QML level. This means you personally are not on the hook to implement any proxy code whatsoever. Contributors would have all the necessary tools to easily add proxy support as needed if they care that much about it. I count 24 filters currently using pixel coordinates that would need a translation shim like the one I demonstrate in the attached ui.qml file. With this method, filter updates can be rolled out as contributors have time. There are no code dependencies that would require a big bang implementation or involvement from you. If you felt generous, you could open up a new “supportsProxy” attribute in the filter Metadatamodel with an icon similar to GPU support to indicate whether a filter supports proxy rescaling. But that’s unnecessary for core functionality.
Extra Bonus:
People post on the forum every week to say their video lags while editing. Proxies could put this problem behind you forever. “Read the FAQ. Over and out.”
Why proxies are worth it (to me at least):
Performance improvement, no matter how much, cannot keep pace with new media formats. The Panasonic S1H camera creates 6K HEVC video files. Two such videos doing a dissolve transition in Shotcut would blow up the CPU with decoding alone. Even iPhones are HEVC now and RED cameras are dropping 8K files, and we haven’t even gotten to AV1 decoding yet. Assuming the CPU didn’t melt from decoding, it would take a monster GPU to apply real-time filters to multi-track 4K+ footage, which would put video editing out of reach for people that didn’t have $3,000+ to spend on a workstation. I don’t think we can buy our way out of this high-def mess with algorithms and hardware. A bigger video file is always around the corner. I feel proxy workflow is the only solution to gracefully handle the huge and CPU-heavy media files created by current and future high-end cameras. Proxy workflow is how my wife and I have edited our 4K videos in Shotcut for the last two years. It actually works very well, and my goal is to finalize it and share it with everyone (with your architectural guidance and blessing). Proxies are a game-changer for multi-track productions that include heavy color grading.
Failed Solutions:
- Generic shims like mlt_rect translation: Filters need to translate much more than Rects. The text filter has a font point size to scale and the blur filters have an amount to scale, while Rotate and Scale have X/Y offsets that aren’t stored in a Rect. I didn’t feel MLT should be responsible for providing a transparent translator for every “coordinate type”. By putting the translation in the UI layer, any random future attribute like font point size can have a custom translator written without modifying MLT or upstream filter libraries.
-
Providing percentage-based sliders on the UI of every pixel-position filter: The problem here, aside from the substantial amount of code rewrite involved, is that users don’t think about coordinates in terms of percentages. If somebody knows X,Y coordinates within an image because they had it open in GIMP and they want to put an effect at that exact X,Y coordinate, a percentage slider makes their job very difficult. If the UI allows both pixel and percentage entry, code gets even more complex because a sync mechanism will be required. Pixel-only entry is intuitive at the UI level, and users have been using it this way for years. We would hate to invalidate all of James Woo’s tutorial screenshots overnight.
-
Storing percentage values in filter storage:
a) If percentages are stored but the UI is in pixels, how are the percentages calculated? If percentages can be calculated, why store them at all?
b) If percentages are stored and the UI is in percents, the pixel values still have to be written because that is what old projects and the filter libraries are hunting for. There would also need to be a translation for old projects from pixels to percents for the UI to work in percents, as well as a sync mechanism to make sure percents are always translated back to pixels for exporting. That’s too much work.
To summarize, I’m trying to make this feature as low-impact on you as possible. Proxies are critical to our 4K workflow, and we’re willing to work to make them happen so long as the plumbing is available. Thanks again for your consideration so far. If you’re really crunched for time, I can submit the necessary tweaks to QmlProducer on GitHub for producer width and height, and then you literally have nothing to do. I figured you would want first dibs on something as architectural as QmlProducer though.
ui.qml.txt (9.8 KB)
I added get(), getDouble(), and getRect() to the QmlProducer for last night’s build so you can continue your work. You can download this under Artifacts from the build server:
http://build.mltframework.org/project.html?projectId=Shotcut&tab=projectOverview
I really appreciate your writeup. I will seriously analyze and consider your approach. The rationale you included is convincing.
This already happens when mlt_property converts the numeric string to an integer when an effect requests as int.
put all translation in the UI QML level.
I was hoping to avoid that to keep this QML UI code simple, but we already lost much of that simplicity when adding keyframes. Making a KeyframableItem that filters can inherit has improved that. We can take the same approach here.
a new “supportsProxy” attribute in the filter Metadatamodel with an icon
The icon shown is mutually exclusive as only one is shown. I have been planning to improve the filter chooser to include category/tags, description, and thumbnail. It could also have emblems that appear for indicating keyframe and proxy support. However, that is probably more work than adding your proxy shim to all the filters that need it.
Some thoughts:
-
Does the Keyframes curves editor need a change? This does not use any logic from the UI QML.
-
This proposal makes things more proxy-ready, but I still want to add features to generate and manage proxy and optimized media before announcing Shotcut does proxy. I am not opposed to working on that.
-
It is strange to see decimal points in values with pixel units. We can put something in the FAQ about that.
-
If you edit the XML outside Shotcut and change the resolution in the profile, it does not change all the filters for you. (Only the first filter on a clip project or Master for a timeline project.) We can require that people editing XML like this need to edit everything in their XML.
-
There is some work being considered to add a lower resolution preview (without generating proxy) that is common in many video editors. This would downscale sources and send a scale factor to all effects. Then, each effect needs to be changed to apply the scale factor. I am not yet sure how this will work with filter bridges such as frei0r and avfilter where some parameters should be scaled and others not. This can be done in a similar manner without doing it all in the engine. Instead of generating proxy, simply change profile.
There is a problem when the profile changes. The UI code only runs when it is loaded and visible in the UI when an item and a filter is selected. The UI contains the recalculation code. We could specify a function signature in Metadata QML. Then, in C++ when profile changes, loop over each filter, locate its QmlMetadata, and invoke this method if it exists. Then, figure out a way to invoke this metadata function from the filter. This needs to be done for every producer in the project. That’s ugly.
That is fantastic. I was away over the weekend, so I will start taking a look at this. Thanks for the fast turn-around!
The pixelToPercent() and recalcScale() functions will always be the same since they are generic translators. Translators for standard structures like Rects could be created using those two base functions. By “inherit”, do you mean put those common translations in a new object similar to producer and profile so that translation code doesn’t take up space in the actual QML file?
I can’t think of a case right off where it would. The nice thing with this method is that filters continue to work as-is using the pixel coordinate system they were written for. For instance, if the Amount parameter of a blur filter had been curve keyframed, the recalcPixelCoords() function steps through the keyframe array translating every Amount value into a rescaled pixel amount. The curve editor will now reflect the new values, and editing could continue as usual. Users would need to be aware that any new manually-keyed coordinates should be in the new profile dimensions, but that should be obvious since they’re the one who changed the profile.
I have you covered there, too. I built a menu-driven DOS batch file system for media management so that my wife can create videos without any involvement from me. I did not create it with the expectation of being bundled with Shotcut. I created it to test and fish out problems in advance so that I could hand you a ready-to-go solution if proxies ever became an integrated piece in Shotcut itself. It has taken over a year to fine-tune the logic to handle pretty much any file thrown at it (although a lot of that has been me learning the nuances of ffmpeg and colorspace signaling or lack thereof). It also builds on the results of extensive format testing to find the best performance and color accurate codecs for this job. The end result supports H.264 All-I, DNxHR HQ, ProRes 422 HQ, HuffYUV, and Ut Video and has full support for YUV or RGB, with or without alpha channel. It also has specialized unsharp masking for each proxy resolution to get the maximum detail possible at proxy size. If you want to take a look, I’ll open up a GitHub repo for you to dig around. What would probably be better is for me to whip up a quick demo video to see if the architecture is even something you want to consider. I’ll start on that just to outline the pitfalls I had along the way and save you some time, regardless of what the final implementation in Shotcut looks like.
The beauty of this method is supposed to be that changes outside of Shotcut would be detected due to the cached dimensions and then filters would automatically reconfigure themselves. I think this lack of filters adjusting themselves is due to your later observation…
So, yeah, that’s devastating…
Rather than register a signature, what if each filter actually had the UI “loaded” so that the Component.onCompleted handler fired? The recalculation happens in onCompleted(), so simply loading the filter would be sufficient, if only for a moment to a UI panel that wasn’t even visible.
I found one other break point too. If a clip has a filter attached with decimal pixels, and the clip is split, then the two resulting clips end up with truncated pixel values. The decimal amounts don’t survive the split.
Yes, I would think that simply changing the profile and letting the filters rewrite themselves would give you this feature implicitly.
Well, this will not be directly usable by Shotcut. I can reuse your findings with regards to formats, codecs, and ffmpeg command lines. However, I need to integrate it with Shotcut’s Jobs and project folder features, and Windows batch file is not cross-platform.
I am not certain that is possible or that it is easy to detect when onCompleted is finished running. It is much easier to invoke a JavaScript function, and all QmlMetadata objects are already held in memory.
I am still thinking about this overall, but I am in a bug-fixing mode at the moment to prepare the 19.07 beta.
For whatever it’s worth, I vote for this in addition to a proxy generator. If full GPU support isn’t happening in Shotcut anytime soon, it would be of great value to give users more than one option to be able to handle work with 4K footage without lag. I don’t know at all if work on a proxy generator would make a lower resolution preview option easier to add but if it does it would be worth aiming for.
The only thing I wonder is how effective would a lower resolution preview option be? I’ve seen videos by some Adobe Premiere Pro users that say that its lower resolution preview doesn’t make much of a difference a lot of the time. Can it be made to be truly effective? If it can, then it would make Shotcut even more appealing as there would be an additional option for those who want to work on 4K footage and beyond but don’t want to wait for proxies to be generated.
This topic was automatically closed after 90 days. New replies are no longer allowed.