The problem is not new with this version; 21.03.21 also has the problem.
Update: I think I did not reboot after having a problem and before running the test on v21.03.21. Yesterday, I rebooted and tried export again with parallel processing turned off, and this time it succeeded with a peak working set of memory usage of 4.5 GB. There were no resource exhaustion events in the Event Viewer system log, but there was one of “Application popup: Windows - Virtual Memory Minimum Too Low” and a little weird activity shortly following. Nevertheless, I could play the video and it passed an integrity check (in Shotcut Properties menu).
It is possible that on one of my tests on v21.05.18 I did not reboot.
Thanks for the investigation! It’s good to know that the error is reproducable. Just to clarify, it works without issues if parallel processing is disabled and without hardware acceleration? I’m trying to get this project rendered as soon as I can since I’ve been having issues with it for months.
I attempted to render using a single core with no hardware acceleration, and although it seemed like it was far more successful, reaching about 30% before my system crashed due to low resources, with melt.exe using about 44 GB of RAM.
C:\Program Files\Shotcut\melt.exe : Current Frame: 1, percentage: 0
At line:1 char:1
+ C:\"Program Files"\Shotcut\melt.exe -progress final-job-2.xml 2>&1 | ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (Current Frame: ...age: 0:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
[mp4 @ 00000198258470c0] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
[mp4 @ 00000198258470c0] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
Current Frame: 2, percentage: 0
Current Frame: 3, percentage: 0
Current Frame: 4, percentage: 0
...
Current Frame: 117, percentage: 0
Current Frame: 118, percentage: 0
Current Frame: 119, percentage: 0
QObject::startTimer: Timers can only be used with threads started with QThread
Current Frame: 120, percentage: 0
Current Frame: 121, percentage: 0
Current Frame: 122, percentage: 0
Current Frame: 123, percentage: 0
[swscaler @ 000001989bd4a700] Warning: data is not aligned! This can lead to a speed loss
Current Frame: 124, percentage: 0
Current Frame: 125, percentage: 0
...
Current Frame: 63451, percentage: 98
Current Frame: 63452, percentage: 98
QImage: out of memory, returning null image
Current Frame: 63453, percentage: 98
Current Frame: 63454, percentage: 98
...
Current Frame: 63534, percentage: 98
Current Frame: 63535, percentage: 98
[consumer avformat] error writing video frame: -12
Current Frame: 63536, percentage: 98
Memory was definitely exhausted. I had about 100 resource exhaustion detections in the event log, and a few processes had some weird behaviour, which could explain the buffer overruns possibly, although I can’t say I know enough about buffers to know what I’m talking about. I imagine that it got further since Shotcut wasn’t open using an extra GB of RAM, so it could leak for a bit longer before crashing.
Thanks for your time, and hopefully we can get to the bottom of this soon! If there’s anything I can do to help find the issue, please let me know, although I doubt my C++ is good enough to get me anywhere.
I was also able to reproduce this using your provided source files. I started the process on a day that I was able to closely monitor the progress. I watched for hours as the melt.exe memory stubbornly hovered around 4GB in the task manager - never exceeding 4.2GB. By 83% complete I was feeling smug that it was not leaking and was surely going to succeed. I looked away and moments later my computer was locked up. After a reboot I find similar virtual memory errors in the event log.
So it would seem that the memory utilization spontaneously spiked in a short period of time. Or maybe the virtual memory usage can accumulate without being reported in the task manager somehow?
If you can afford the time, a time consuming task that I would find helpful…
It would be good if we could reproduce the problem with a simpler project file. I wonder if it is only the length of the project causing the problem. Or maybe the source files exploit a problem. Or maybe too many tracks.
If you could try subtracting features from the project one-by-one and see if it still fails for you, that would help narrow where the problem could be. For example, if the problem can be reproduced without using the timeline, that would eliminate a large swath of areas for investigation.
To see the virtual memory used in Task Manager Details view, show the column “Commit size” This you can see goes up, but I do not yet know how it trends over time. Usually, this takes so long I have to let it run overnight, or I leave the house for an extended period.
Update:
I have been running some tests outside of the project and not yet reproduced a virtual memory consumption problem. I am also doing some memory profiling with the project but not found anything. All this stuff takes a long time. Then, I starting giving some attention to the frequent error messages “moov atom not found.” The copy of render particles mask.mp4 is not valid and not opening in Shotcut, ffmpeg, or anything else. I downloaded it again from the Google drive, and it is the same. I suspect this could be a factor in the problem.
That’s very interesting - I’ll have a closer look at that one after my classes today. I did do some manual editing of the .mlt file a few months back when I accidentally broke the project and replaced the actual clips with the proxy clips, but the preview seems to work fine.
I’ll try a trimmed version of the project as well then. Is there a command line option to get melt to print out timestamps every time it updates the progress? That way I could pair it with some form of memory monitoring to figure out at what frame the issues start.
Yeah I’ve got that to work, I’m more looking for a thing where it’ll print something like [@t=2021/06/07 11:56:42] Current frame: 2534, percentage: 18.
The problem is related to the Size, Position & Rotate filters. If I remove them the problem goes away, if I move them from the track to the clips, it does not help. I did intensive memory profiling of the first couple seconds with a special tool called valgrind, but it did not reveal anything. It runs many times slower under this tool such that trying to let it progress to over 70% might take days, and then it may not reveal anything about a pure consumption issue (it is designed more for hard leaks).
I am running another test outside of the project with a 2 hour 4K video with the filter applied to zoom in 140%, and so far at 37% done the virtual memory (commit size) usage in Task Manager is not showing anything odd (flat at about 500MB more than active private working set is OK).
Oh cool! I was thinking about how I could get valgrind to work on it (with the concern being that I don’t think valgrind works properly if you terminate the program prematurely, meaning that you can’t just hit ctrl+c on melt as soon as it starts playing up). I can’t say I’m familiar with valgrind enough to be certain of the behaviour though.
Thanks so much for the investigation though - I’ll muck around with trying to create a smaller project that still reproduces the issue later tonight, just have a few other things to sort out first.
I’ve had a look, but sadly, I haven’t been successful in making a smaller project that reproduces the errors, although I haven’t had time to try every video track individually first. I’ll keep looking, and let you guys know when I find something. Are there any updates on anyone else’s end?
I am marking this bug as solved because I believe it is a duplicate of this issue.
It has the same composition: A frei0r filter before an affine filter. If anyone wants to retest this using the latest build, please feel free to report back.
