Shotcut Color Accuracy

[chris319]

Hi DRM -

Thank you for your interest.

Like I said before how about integrating it into Shotcut? I know you said it seems daunting but you could take your time in doing it. It’s not like there is a deadline for contributing code to Shotcut.

Mind you, I’m testing these codecs apart from Shotcut. I’m able to get accurate colors by talking directly to ffmpeg, but I don’t know what Shotcut is doing to colors under the hood. It would be nice if Shoutcut could export video without color errors, and I have proven that it is possible in theory, at least with x264, ffv1 and Huffyuv. In addition to coding there is the whole issue of testing, basically doing Q.C. on the code that’s been written.

WRT writing code for Shotcut, I would be starting at the very bottom of the learning curve and would require a great deal of handholding from the Shotcut developers to figure out what’s going on. Not only is there Shotcut by MLT as well. Without my writing new code I think the best idea would be to fix the color errors in Shotcut and to address issues in the color grading UI.

I’m building my program to professional standards and won’t write code that isn’t up to professional standards. Whether this is in keeping with Shotcut’s basic philosophy, I don’t know.

I started to tool up to compile Shotcut some time ago and forget how far I got. Fortunately, I didn’t remove any of it from my computer. So I might give it another shot with an eye toward addressing Shotcut’s existing color issues.

I haven’t even looked at Shotcut’s new scope feature.

One question I have posed and have not received an answer to is whether Shotcut addresses individual pixels.

Here is my bug report that the ffmpeg people blew off:

https://trac.ffmpeg.org/ticket/6924

[chris319]

This from the Shotcut compilation how-to:

The section above just gets you a working environment for the Shotcut code, but not any of its dependencies. Some of the dependencies included in the SDK with source code can be opened and used within Qt Creator - frei0r and webvfx. Instructions for that are not yet included.

“Instructions not yet included”. Already this sounds like an uphill battle. If it’s going to be more work setting up the development environment than it will be fixing problems in the actual program, I’m not interested. And I have to install the GitBash shell on top of everything else?

I’ve been writing my code in PureBasic. With it you basically install PureBasic, write your code and press F5 — much simpler.

Maybe my program has some use as a transcoder? Or as a color error checker?

[Paul2]

@chris319

Had a look at your bug report to the ffmpeg devs.
Few points:

1. Is it possible to make the original screenshots that you referred to available again? The links are broken so it’s hard to get the full picture (no pun intended).

2. In the bug report, you also mentioned a scope display, again no picture.
   What scope did you use?
   Hardware, if so are you 100% sure that the video levels feeding it were not introducing any errors?
   If software, which software and how accurate is it?

Reason I’m asking is that I tried the -vf scale=in_range=pc:out_range=tv in ffmpeg and it worked perfectly.

Below a screenshot of waveform, parade and vector for the original video with levels from 0-1024, yes I mostly use 10 bit video but for 8 bit video, divide by 4, 0-255

After using the -vf scale in ffmpeg, the video is indeed limited as per rec709 (64-960 for 10 bit or 16-235 for 8 bit):

What version of ffmpeg are you using?
Another way is to use a LUT, Shotcut supports 3d LUTS in .cube format.

[chris319]

I wrote the scope code myself. The calculations are not complicated: the x axis of the picture vs pixel luminance on the y axis.

The scope code I was using at the time that bug report was filed, was flawed. I was obtaining RGB data from a bad source and calculating the luminance. GIGO.

I have since rewritten the code to either use RGB samples from an RGB file and calculate Y, or to read Y directly from a YUV file. It has been tested and compared to the ffmpeg scope and was determined to be accurate. It also has IRE units which professionals still use. I’ll see about reconstituting those scope shots.

Do you know exactly how ffmpeg goes about confining the video levels to the 16-235 range?

EBU r 103 is more lenient than BT.709. It makes allowance for ringing and overshoots up to 246 and down to 5.

[Paul2]

@chris319

Do you know exactly how ffmpeg goes about confining the video levels to the 16-235 range?

Nope but the source is available.

[shotcut]

@chris319

It would be nice if Shoutcut could export video without color errors

I already explained to you how you are generating clips at full range without signaling it. Of course, that is going to lead to incorrect results as I already demonstrated in the other thread on intermediate files. Until you fix that, your tests are invalid, and you need to stop making claims without evidence. Meanwhile, I provided some counter evidence to this claim in that thread. Of course, it is not perfect, but it is much better than what you were reporting while using a faulty process.

whether Shotcut addresses individual pixels

It depends on what you mean by “Shotcut”. Strictly speaking, no; except in preview - when not using GPU Effects - it receives YUV from MLT and uses an OpenGL fragment shader to render in RGB. If you are referring to the whole build with FFmpeg and MLT libraries included, then, of course.

[chris319]

@shotcut

you need to stop making claims without evidence

See posts 21, 22, 55, 57, 58, 70, 71 and 75 in the thread you referred to. There are plenty of test results and evidence in those eight posts.

I just did another test using Shotcut’s internal color generator, not my own file, it’s all Shotcut with nothing external involved to generate the output file and the Shotcut defaults for Lossless H.264. Here is the Media Info data on the file created and output by Shotcut:

General
Complete name                            : D:\Videos\SonyAX53\Color tests.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 6.68 MiB
Duration                                 : 10 min 0 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 93.4 kb/s
Writing application                      : Lavf58.12.100

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High Intra@L4
Format settings                          : CABAC
Format settings, CABAC                   : Yes
Format settings, GOP                     : N=1
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 10 min 0 s
Bit rate mode                            : Variable
Bit rate                                 : 86.6 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.002
Stream size                              : 6.19 MiB (93%)
Writing library                          : x264 core 155 r2917 0a84d98
Encoding settings                        : cabac=1 / ref=1 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=12 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=0 / weightp=0 / keyint=1 / keyint_min=1 / scenecut=40 / intra_refresh=0 / rc=crf / mbtree=0 / crf=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Color range                              : Limited
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
Codec configuration box                  : avcC

Here is the Color Range line from Media Info:

]Color range : Limited

Here is where my color checking program interfaces with ffmpeg to import a frame of video:

pipeIn$ = "-i " + filename$ + " -f image2pipe  -s 1920x1080   -pix_fmt yuv420p  -r 25  -vcodec rawvideo -"

Here is where my code converts YUV to RGB:

]Pr.f = (vf -128.0)/ 255
Pb.f = (uf-128.0) / 255
Yf.f / 255
Rf.f=Yf+Pr*2*(1-#Kr)
Gf.f=Yf-(2*Pb*(1-#Kb)*(#Kb/(1-#Kr-#Kb)))-(2*Pr*(1-#Kr)*(#Kr/(1-#Kr-#Kb))) 
Bf.f=Yf+Pb*2*(1-#Kb)
If rf > 1: rf = 1:EndIf: If gf > 1: gf = 1:EndIf: If bf > 1: bf = 1:EndIf
If rf < 0: rf = 0:EndIf: If gf < 0: gf = 0:EndIf: If bf < 0: bf = 0:EndIf

rf * 255
gf * 255
bf * 255

Using this same code I am getting colors accurate to within rounding error by talking directly to ffmpeg as measured by this and another “eyedropper” program. This program reads YUV values directly from a file; the eyedropper reads RGB values from the screen.

[shotcut]

Again, most of your comments in that thread were based on a full range clip without proper signaling or an unspecified input. Also, I already mentioned and agreed there are issues with the color generator, and I will address this in the next version. All your negative results so far appear to be using known-flawed, niche test vectors. Use your bmp, a full range clip with a yuvj pixfmt, or a limited range yuv-without-j pixfmt clip as your test material. Those probably represent 99% of what 99% of users are using. I do want to address problems, but there needs to be more coherence in the test process and problem reports. I will also add an override for the full range signaling in the Properties panel in the next version to help with those problem clips.

[chris319]

@shotcut

Use your bmp, a full range clip with a yuv j pixfmt, or a limited range yuv-without-j pixfmt clip as your test material. Those probably represent 99% of what 99% of users are using.

As I said in the post just prior, I am using Shotcut’s color generator and export function with default settings. Everything is entirely within Shotcut. None of my images are being used.

The Sony camcorder I use puts out YUV420P, not YUVj. By default, Shotcut outputs YUV420 pixels in the H.264 presets I’ve tried.

[shotcut]

@chris319

I am using Shotcut’s color generator

Sometimes; sometimes not. Sometimes you say you are using a BMP, a clip generated with ffmpeg, or unspecified in your posts. Anyways, this is just talking in circles.

The Sony camcorder I use puts out YUV420P

1. Start with (16,180,16) BMP:
   

2. Generate limited.mp4 with ffmpeg:
   ffmpeg -y -loop 1 -t 10 -r 59.94 -s 1280x720 -i r16g180b16-shotcut-rgb24.bmp -vf zscale=matrix=709:range=limited,format=yuv420p -c:v libx264 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -crf 0 -an limited.mp4

3. Verify ffmpeg output by converting it back to BMP and inspect it with Paint:
   ffmpeg -i limited.mp4 -t 0.016 -y limited.bmp
   

Not great, but provides a baseline against which to compare Shotcut’s output.

4. Open limited.mp4 into Shotcut v18.11.18, and export it using the lossless H.264 preset:
   

   

5. Use ffmpeg to convert one frame of limited-shotcut-yuv420p.mp4 to BMP, and inspect the BMP in Paint:
   

Exact same result as using ffmpeg directly.

[chris319]

@shotcut

Use your bmp, a full range clip with a yuv j pixfmt, or a limited range yuv-without-j pixfmt clip as your test material. Those probably represent 99% of what 99% of users are using.

Basically you’re saying that Shotcut works in limited-range yuv without the “j” and all testing should be done in limited-range, correct?

[chris319]

Today I installed Shotcut v.18.11.18

COLOR TESTS:

Input: bmp file: 16-180-16
bmp file encoded to mp4 with ffmpeg LIMITED RANGE H.264
Test mp4 file: 16-179-14
Shotcut preview window: 17-179-16

Test mp4 file encoded with Shotcut lossless codecs:

H.264: 16-179-14
FFV1: 15-178-13
HuffYUV: 15-178-13
UtVideo: 29-208-20

Change in testing methodology: viewing files with Windows Media Player and checking colors with eyedropper program.

All lossless codecs and Shotcut preview window within rounding error except UtVideo.

Thank you, Dan, for your attention to color accuracy.

[chris319]

Re: UtVideo, my thinking is that the color error is a UtVideo codec issue rather than a Shotcut issue, so maybe the UtVideo adherents want to bring this up with the developer of UtVideo.

[DRM]

So if I got this right on v.18.11.18 all codecs have the colors now corrected with the exception of Ut Video? I have grown fond of Ut Video ever since @Austin thankfully introduced it on the forums and Dan graciously added the presets to it for last version’s Shotcut so it’s disappointing to hear there are issues with it. If there are color problems then can Ut Video’s colors be corrected for the next version, @shotcut?

I was looking up some info for Ut Video and I came across this post on another forum

Seven Mathematically Lossless Codecs (legacy article)

I’m not technical at all @chris319 but maybe this has some relevance regarding this topic?

[chris319]

@DRM

So if I got this right on v.18.11.18 all codecs have the colors now corrected with the exception of Ut Video?

No. The lossless codecs HuffYUV, FFV1 and H.264. I haven’t tested ProRes or other lossy codecs yet but I’ll add ProRes to the list.

As I said in a prior post, UtVideo’s color problems probably have to be brought up with the developer as it is likely a codec problem and not a Shotcut problem.

[DRM]

@chris319

As I said in a prior post, UtVideo’s color problems probably have to be brought up with the developer as it is likely a codec problem and not a Shotcut problem.

If the problem is with the codec itself is there anything Shotcut can be programmed to do in order to correct it on its end?

And thanks for testing out Ut Video by the way.

[chris319]

@DRM

If the problem is with the codec itself is there anything Shotcut can be programmed to do in order to correct it on its end?

Likely not.

[Austin]

@shotcut, this post contains a solution that makes the Ut Video preset color-accurate. Would you be willing to update the preset for the next version of Shotcut? I really appreciate your efforts at making the colors look so good!

After exporting around 200 test files, I finally understand what’s happening with the Ut Video color issues, and I’ve found simple and predictable workarounds for the common scenarios. Basically, there’s nothing wrong with Ut Video as a codec. The problem is colorspace signaling within ffmpeg. Here are the scenarios and their workarounds using Shotcut 18.11.18 and ffmpeg 4.0.2, tested on Windows 10 and Linux Mint 18.3:

Scenario: Exporting a video from Shotcut using the Ut Video preset

This doesn’t work:
Simply clicking the Ut Video preset and exporting.

This does work:
Go to the Export > Advanced > Other tab and add these lines:
color_range=mpeg
pix_fmt=yuv422p
Then export as usual. @shotcut, I believe adding these lines to the preset would cover 99.9% of use cases. No one should mind the change since it wasn’t working in the first place.

The problem:
If nothing is added to the Other tab, MediaInfo reports that the Ut Video file was encoded in RGB. That’s a massive problem because the Lossless H.264 source file was YUV444. (I used the limited.mp4 file as generated in previous posts.) I chose yuv422p with MPEG range as the export format because that’s the native internal processing format of Shotcut (best I recall). I didn’t see a point in making the preset use yuv444p when Shotcut won’t deliver 4:4:4 material to it in the first place. Likewise, using yuv420p for the preset didn’t make sense because Ut Video is for people who want to get as lossless as possible. (I tested 4:2:0 to 4:2:2 to 4:4:4 for conversion errors, and there were none.) Curiously, it was not necessary to specify a colorspace in the Other box. I assume Shotcut is explicitly passing this to ffmpeg since Shotcut knows the colorspace from the video mode profile. In fact, adding colorspace switches to the Other box actually messed up the colors rather than fix them. Perhaps a double conversion happened in that case.

Scenario: Transcoding a video with ffmpeg command line

This doesn’t work:
ffmpeg -i limited.mp4 -c:v utvideo -an export.avi

This does work:
ffmpeg -i limited.mp4 -colorspace 1 -color_primaries 1 -color_trc 1 -c:v utvideo -an export.avi

The problem:
ffmpeg figures out (or defaults) to BT.709 colorspace upon reading the limited.mp4 file. It also figures out that the file is in YUV444 format with MPEG range and correctly passes this to Ut Video. However, ffmpeg does not pass the BT.709 spec to Ut Video (which is a bug), so Ut Video defaults itself to BT.601 colorspace. The mismatch shifts the colors, and BT.601 is confirmed because MediaInfo says the FourCC is ULY4 (that’s Ut Video-speak for “4:4:4 8-bit limited BT.601”). Adding switches to explicitly declare the colorspace will push Ut Video into BT.709 and then it works. I can’t find documentation confirming that “1” as a colorspace value means “explicitly tag the output file with the same colorspace as the input”, but this seems to be how it works. I would rather be generic than hard-code “bt709” in my batch conversion scripts, just in case a source actually is bt601. @DRM, you and I will need to keep this workaround in mind if using ffmpeg scripts to create Ut Video proxies.

Scenario: Using ffmpeg to create video from an image file

This doesn’t work:
ffmpeg -loop 1 -t 5 -r 29.97 -s 1920x1080 -i r16g180b16-shotcut-rgb24.bmp -vf zscale=matrix=709:range=limited,format=yuv444p -c:v utvideo -an limited.avi

This does work:
ffmpeg -loop 1 -t 5 -r 29.97 -s 1920x1080 -i r16g180b16-shotcut-rgb24.bmp -vf zscale=matrix=709:range=limited,format=yuv444p -colorspace 1 -color_primaries 1 -color_trc 1 -c:v utvideo -an limited.avi

The problem:
This is the same issue as above. The filter graph is in BT.709 mode, but ffmpeg isn’t communicating that fact to the Ut Video codec. So again, Ut Video defaults itself to BT.601 colorspace which is confirmed by MediaInfo. Explicitly declaring the colorspace (even if using “1” for pass-thru) solves this problem.

I think I have enough test cases (around 40) to submit a bug report to ffmpeg. If anyone else could confirm or refute this hypothesis before I send it in, that would be awesome. In particular, my tests are very incomplete when the source material actually is in BT.601 colorspace. It would be great if one conversion script could handle a source in any colorspace or pixel format (bt601, bt709, RGB, RGBA, etc).

So, figuring out this madness is how I burned my afternoon yesterday. How about you guys? This Shotcut forum is a thing of beauty. A conversation with @D_S got me aware of Ut Video’s existence in the first place and it quickly replaced my H.264 CRF 12 proxies. Then @chris319 found a color problem that would have compromised my hopes of color grading on the proxies. Now it’s fixed, and on the audio side @shotcut informed me that AC-3 doesn’t have the delay issues that AAC does. @DRM got me thinking about proxy resolution again, and as I’ll describe on the proxy thread soon, I’m moving away from 480x270 because 270 isn’t divisible by 4 (long story). After pooling it all together, this new proxy setup is an editing dreamland and I doubt I would have found it on my own. Thanks everyone for the ideas!

[shotcut]

@Austin

Would you be willing to update the preset for the next version of Shotcut?

Yes

[shotcut]

@Austin

I chose yuv422p with MPEG range as the export format because that’s the native internal processing format of Shotcut (best I recall).

It depends on “mlt_image_format” in the Other tab (consumer property in MLT terminology) and if certain filters or transitions are added that must convert to YUV for their job (a minority). One could also try using mlt_image_format=rgb24 or rgb24a with the Ut Video presets to workaround this problem.