NVENC lossless export results in corrupt video

captapraelium · July 12, 2018, 6:48am

Bear with me, this is an odd one…

I have recorded footage from a game, recorded by OBS Studio using NVENC. Watching that video there are no issues.

If I create a new shotcut project, add that file to the playlist, and export it using NVENC with lossless parameters, the output file is corrupted. There are no issues with x264 using the same/equivalent parameters. Here is a screenshot of what it looks like: https://imgur.com/a/ES8QcxT

If I watch the video in MPC or VLC, parts of the video become garbled, blocky, smeared colours, it’s like it’s dropping frames - but it isn’t (the dropped frames shown in the above screenshot are from skipping to the affected part of the video). It’s always the same part of the same videos. I noticed that the current bitrate skyrockets to over double the average, in the effected areas. This is not occurring with all of my recorded footage, only certain files.

If I watch it in shotcut itself, there is no corruption visible. If I upload the file to youtube, the corruption is not visible.

Reducing the quality of the export improves the corruption, until it becomes invisible at around 80%.

I’m at a loss here. Any advice?

captapraelium · July 16, 2018, 9:37am

Just to follow up: This goes way upstream to nvidia. About a week of all-day testing, by first converting my shotcut project to a melt command line, stripping it down to the input file alone, converting that to an ffmpeg command line, testing with raw yuv input, has led me to confirm that constqp encodes with qp<5 will fail with certain input (generally, high movement scenes). Using the newer lossless preset works fine for quality above that.

This does present a few issues with shotcut however.

Firstly, the quality slider is translated into the global_quality parameter for mlt, however that parameter is deprecated for use with nvenc. Not really an immediate issue, but soon…

Secondly, once that slider goes up past the equivalent for qp=5 / vglobal_quality 5, there is a likelihood of corruption in the output.

Thirdly, it’s also translated into the ‘vq=’ option, and I can’t find any documentation for that anywhere…

And last of all the constqp mode is always used, even if we set another rate control method in our additional parameters.

It seems like maybe the export dialog could use some work to allow it to be configurable for different encoders and options. Thoughts?

shotcut · July 17, 2018, 11:24pm

No plans to put that level of nuance into the code for nvenc.

In MLT the ‘v’ prefix on an avformat consumer property translates into “only for a video stream or codec context”. In FFmpeg, ‘q’ is “use fixed quality scale” and interpretation depends on codec. Sometimes, it may not be used by the codec, but it facilitates Shotcut ability to interpret presets.

No, it is not on the latest version 18.07 at least. I seem to recall a bug like this on earlier version.

Yes, it could. That can also result in a massive amount of code to maintain for every combination of muxer, codec, and rate control mode. I chose not to add crazy amounts of code that will introduce its own bugs and be difficult to maintain especially for new FFmpeg versions. Instead, some things are handled generically (FFmpeg used to make greater usage of common options) and there is some code for popular codecs. Are you volunteering?

captapraelium · July 22, 2018, 10:20am

Agreed, nuancing the code for the myriad different output configurations would be unwieldly and awful to maintain. I understand that FFmpeg is certainly a moving goalpost. Please, don’t get me wrong here, I can see what you’re doing and why you’re doing it and I agree that it is the best way. I just feel it needs a few tweaks is all. I would not be helping the project by failing to mention this. Naturally it is best to have a ‘just works for most people’ setup as we have now with x264… However it strikes me that the default options should not override options specified by the user.

I am using 18.07.02 and this is the output XML: Note the constqp mode which is always present regardless of the mode I select by manually specified options -

<consumer vcodec="h264_nvenc" bf="2" vq="21" ab="384k" sc_threshold="0" r="60" target="C:/Users/Capta/Videos/test.mkv" cq="15" width="2560" height="1440" progressive="1" rc="constqp" vglobal_quality="21" acodec="aac" channels="2" rescale="hyper" ar="96000" strict_gop="1" deinterlace_method="yadif" profile="2" f="matroska" aspect="1.77778" real_time="-4" g="30" preset="1" qp="15" top_field_first="2" threads="11" mlt_service="avformat"/>

In this example, my Export…‘Other’ tab contains the following:
preset=1 profile=2 rc=9 cq=15 qp=15 strict_gop=1

rc=9 aka vbr_hq is ignored and overridden as constqp aka rc=0.

I imagine it should not be overly difficult to ensure that options specified in the ‘Other’ tab, which are equivalent or identical to default options (such as rc=rc, global_quality=vq=qp, etc), are given priority over those default options. Failing this, it would be better to pass minimal or even no options at all, simply using the ffmpeg defaults unless manually specified. Please stop me if you disagree - I just feel that it would be better to have something which required user interaction than something which forcibly broke the encoder.

Yes, I am volunteering. I have extensive experience in C and C++ professionally and am familiar with code versioning systems however I’ve never had a need to use github and would prefer not to spend a week reverse engineering MLT again Are you able to help me get into this? This is my gaming and video editing PC, it is not and would not be suited to be used for dev work. Is there a VM configured to dev shotcut?

shotcut · July 22, 2018, 4:35pm

There is this:
https://shotcut.org/notes/windowsdev/

As you can see here, the only time constqp is set is when Quality-based VBR is chosen:

github.com

mltframework/shotcut/blob/master/src/docks/encodedock.cpp#L601


    break;
case RateControlConstant: {
    const QString& b = ui->videoBitrateCombo->currentText();
    p->set("cbr", 1);
    p->set("vb", b.toLatin1().constData());
    p->set("vminrate", b.toLatin1().constData());
    p->set("vmaxrate", b.toLatin1().constData());
    p->set("vbufsize", int(ui->videoBufferSizeSpinner->value() * 8 * 1024));
    break;
    }
case RateControlQuality: {
    int vq = ui->videoQualitySpinner->value();
    p->set("rc", "constqp");
    p->set("vglobal_quality", TO_ABSOLUTE(51, 0, vq));
    p->set("vq", TO_ABSOLUTE(51, 0, vq));
    break;
    }
case RateControlConstrained: {
    const QString& b = ui->videoBitrateCombo->currentText();
    int vq = ui->videoQualitySpinner->value();
    p->set("qmin", TO_ABSOLUTE(51, 0, vq));

captapraelium · July 25, 2018, 1:28am

Hmm, it seems I’m not allowed to use the same windows license in a VM. Is it possible to cross-compile from linux? That way I can use a linux VM. Geez I miss my VLK when I’m home XD

Interesting that you are already handling nvenc as it’s own case.
} else if (vcodec.contains("nvenc")) {

It might be more straightforward to just create new RateControl* cases which allow for the desired functionality (such as CRF) but then we create more moving parts to maintain, and that only really solves my own direct issues with nvenc and not any other related issues that may arise. I’m not sure the direction you’d prefer to take here.

Still, having user-specified parameters override those from the GUI would be most flexible. It seems that the source of this is the handling of CollectProperties around line 486, it first reads from the advanced section and then from the GUI, so the GUI will overwrite advanced entries. Probably should go the other way around? (ie read from the GUI first then overwrite that with anything from the advanced textbox). I notice around line 1336 there is actually a workaround for this. Correct me if I’m wrong this is just a first glance.

shotcut · August 7, 2018, 6:56pm

For the next version 18.09, I added a function setIfNotSet() called by collectProperties() to let one override any generated option in the Other tab. There is one exception: the x265-params. For that, as was always the case, the generated key-value pairs are prepended to an existing x265-params in Other. You can still override x265-params; it is just a little less clean and elegant (always been this way).

captapraelium · August 14, 2018, 2:08pm

Sounds great! Hyped for .09 Thanks man! I’ll be sure to test it out and let you know how it goes.

shotcut · August 14, 2018, 6:23pm

Actually, this change went into v18.08.11 as well.