Shotcut Color Accuracy

chris319 · November 24, 2018, 2:24pm

See posts 21, 22, 55, 57, 58, 70, 71 and 75 in the thread you referred to. There are plenty of test results and evidence in those eight posts.

I just did another test using Shotcut’s internal color generator, not my own file, it’s all Shotcut with nothing external involved to generate the output file and the Shotcut defaults for Lossless H.264. Here is the Media Info data on the file created and output by Shotcut:

General
Complete name                            : D:\Videos\SonyAX53\Color tests.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 6.68 MiB
Duration                                 : 10 min 0 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 93.4 kb/s
Writing application                      : Lavf58.12.100

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High Intra@L4
Format settings                          : CABAC
Format settings, CABAC                   : Yes
Format settings, GOP                     : N=1
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 10 min 0 s
Bit rate mode                            : Variable
Bit rate                                 : 86.6 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.002
Stream size                              : 6.19 MiB (93%)
Writing library                          : x264 core 155 r2917 0a84d98
Encoding settings                        : cabac=1 / ref=1 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=12 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=0 / weightp=0 / keyint=1 / keyint_min=1 / scenecut=40 / intra_refresh=0 / rc=crf / mbtree=0 / crf=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Color range                              : Limited
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
Codec configuration box                  : avcC

Here is the Color Range line from Media Info:

]Color range : Limited

Here is where my color checking program interfaces with ffmpeg to import a frame of video:

pipeIn$ = "-i " + filename$ + " -f image2pipe  -s 1920x1080   -pix_fmt yuv420p  -r 25  -vcodec rawvideo -"

Here is where my code converts YUV to RGB:

]Pr.f = (vf -128.0)/ 255
Pb.f = (uf-128.0) / 255
Yf.f / 255
Rf.f=Yf+Pr*2*(1-#Kr)
Gf.f=Yf-(2*Pb*(1-#Kb)*(#Kb/(1-#Kr-#Kb)))-(2*Pr*(1-#Kr)*(#Kr/(1-#Kr-#Kb))) 
Bf.f=Yf+Pb*2*(1-#Kb)
If rf > 1: rf = 1:EndIf: If gf > 1: gf = 1:EndIf: If bf > 1: bf = 1:EndIf
If rf < 0: rf = 0:EndIf: If gf < 0: gf = 0:EndIf: If bf < 0: bf = 0:EndIf

rf * 255
gf * 255
bf * 255

Using this same code I am getting colors accurate to within rounding error by talking directly to ffmpeg as measured by this and another “eyedropper” program. This program reads YUV values directly from a file; the eyedropper reads RGB values from the screen.

shotcut · November 24, 2018, 9:18pm

Again, most of your comments in that thread were based on a full range clip without proper signaling or an unspecified input. Also, I already mentioned and agreed there are issues with the color generator, and I will address this in the next version. All your negative results so far appear to be using known-flawed, niche test vectors. Use your bmp, a full range clip with a yuvj pixfmt, or a limited range yuv-without-j pixfmt clip as your test material. Those probably represent 99% of what 99% of users are using. I do want to address problems, but there needs to be more coherence in the test process and problem reports. I will also add an override for the full range signaling in the Properties panel in the next version to help with those problem clips.

chris319 · November 24, 2018, 9:35pm

As I said in the post just prior, I am using Shotcut’s color generator and export function with default settings. Everything is entirely within Shotcut. None of my images are being used.

The Sony camcorder I use puts out YUV420P, not YUVj. By default, Shotcut outputs YUV420 pixels in the H.264 presets I’ve tried.

shotcut · November 24, 2018, 10:22pm

Sometimes; sometimes not. Sometimes you say you are using a BMP, a clip generated with ffmpeg, or unspecified in your posts. Anyways, this is just talking in circles.

The Sony camcorder I use puts out YUV420P

Start with (16,180,16) BMP:
Generate limited.mp4 with ffmpeg:
ffmpeg -y -loop 1 -t 10 -r 59.94 -s 1280x720 -i r16g180b16-shotcut-rgb24.bmp -vf zscale=matrix=709:range=limited,format=yuv420p -c:v libx264 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -crf 0 -an limited.mp4
Verify ffmpeg output by converting it back to BMP and inspect it with Paint:
ffmpeg -i limited.mp4 -t 0.016 -y limited.bmp

Not great, but provides a baseline against which to compare Shotcut’s output.

Open limited.mp4 into Shotcut v18.11.18, and export it using the lossless H.264 preset:

image1357×881 31.1 KB
Use ffmpeg to convert one frame of limited-shotcut-yuv420p.mp4 to BMP, and inspect the BMP in Paint:

Exact same result as using ffmpeg directly.

chris319 · November 25, 2018, 3:48am

Basically you’re saying that Shotcut works in limited-range yuv without the “j” and all testing should be done in limited-range, correct?

chris319 · November 25, 2018, 5:53pm

Today I installed Shotcut v.18.11.18

COLOR TESTS:

Input: bmp file: 16-180-16
bmp file encoded to mp4 with ffmpeg LIMITED RANGE H.264
Test mp4 file: 16-179-14
Shotcut preview window: 17-179-16

Test mp4 file encoded with Shotcut lossless codecs:

H.264: 16-179-14
FFV1: 15-178-13
HuffYUV: 15-178-13
UtVideo: 29-208-20

Change in testing methodology: viewing files with Windows Media Player and checking colors with eyedropper program.

All lossless codecs and Shotcut preview window within rounding error except UtVideo.

Thank you, Dan, for your attention to color accuracy.

chris319 · November 26, 2018, 12:47am

Re: UtVideo, my thinking is that the color error is a UtVideo codec issue rather than a Shotcut issue, so maybe the UtVideo adherents want to bring this up with the developer of UtVideo.

DRM · November 26, 2018, 6:58pm

So if I got this right on v.18.11.18 all codecs have the colors now corrected with the exception of Ut Video? I have grown fond of Ut Video ever since @Austin thankfully introduced it on the forums and Dan graciously added the presets to it for last version’s Shotcut so it’s disappointing to hear there are issues with it. If there are color problems then can Ut Video’s colors be corrected for the next version, @shotcut?

I was looking up some info for Ut Video and I came across this post on another forum

I’m not technical at all @chris319 but maybe this has some relevance regarding this topic?

chris319 · November 26, 2018, 9:03pm

No. The lossless codecs HuffYUV, FFV1 and H.264. I haven’t tested ProRes or other lossy codecs yet but I’ll add ProRes to the list.

As I said in a prior post, UtVideo’s color problems probably have to be brought up with the developer as it is likely a codec problem and not a Shotcut problem.

DRM · November 26, 2018, 9:24pm

If the problem is with the codec itself is there anything Shotcut can be programmed to do in order to correct it on its end?

And thanks for testing out Ut Video by the way.

chris319 · November 26, 2018, 9:29pm

Likely not.

Austin · November 28, 2018, 6:11pm

@shotcut, this post contains a solution that makes the Ut Video preset color-accurate. Would you be willing to update the preset for the next version of Shotcut? I really appreciate your efforts at making the colors look so good!

After exporting around 200 test files, I finally understand what’s happening with the Ut Video color issues, and I’ve found simple and predictable workarounds for the common scenarios. Basically, there’s nothing wrong with Ut Video as a codec. The problem is colorspace signaling within ffmpeg. Here are the scenarios and their workarounds using Shotcut 18.11.18 and ffmpeg 4.0.2, tested on Windows 10 and Linux Mint 18.3:

Scenario: Exporting a video from Shotcut using the Ut Video preset

This doesn’t work:
Simply clicking the Ut Video preset and exporting.

This does work:
Go to the Export > Advanced > Other tab and add these lines:
color_range=mpeg
pix_fmt=yuv422p
Then export as usual. @shotcut, I believe adding these lines to the preset would cover 99.9% of use cases. No one should mind the change since it wasn’t working in the first place.

The problem:
If nothing is added to the Other tab, MediaInfo reports that the Ut Video file was encoded in RGB. That’s a massive problem because the Lossless H.264 source file was YUV444. (I used the limited.mp4 file as generated in previous posts.) I chose yuv422p with MPEG range as the export format because that’s the native internal processing format of Shotcut (best I recall). I didn’t see a point in making the preset use yuv444p when Shotcut won’t deliver 4:4:4 material to it in the first place. Likewise, using yuv420p for the preset didn’t make sense because Ut Video is for people who want to get as lossless as possible. (I tested 4:2:0 to 4:2:2 to 4:4:4 for conversion errors, and there were none.) Curiously, it was not necessary to specify a colorspace in the Other box. I assume Shotcut is explicitly passing this to ffmpeg since Shotcut knows the colorspace from the video mode profile. In fact, adding colorspace switches to the Other box actually messed up the colors rather than fix them. Perhaps a double conversion happened in that case.

Scenario: Transcoding a video with ffmpeg command line

This doesn’t work:
ffmpeg -i limited.mp4 -c:v utvideo -an export.avi

This does work:
ffmpeg -i limited.mp4 -colorspace 1 -color_primaries 1 -color_trc 1 -c:v utvideo -an export.avi

The problem:
ffmpeg figures out (or defaults) to BT.709 colorspace upon reading the limited.mp4 file. It also figures out that the file is in YUV444 format with MPEG range and correctly passes this to Ut Video. However, ffmpeg does not pass the BT.709 spec to Ut Video (which is a bug), so Ut Video defaults itself to BT.601 colorspace. The mismatch shifts the colors, and BT.601 is confirmed because MediaInfo says the FourCC is ULY4 (that’s Ut Video-speak for “4:4:4 8-bit limited BT.601”). Adding switches to explicitly declare the colorspace will push Ut Video into BT.709 and then it works. I can’t find documentation confirming that “1” as a colorspace value means “explicitly tag the output file with the same colorspace as the input”, but this seems to be how it works. I would rather be generic than hard-code “bt709” in my batch conversion scripts, just in case a source actually is bt601. @DRM, you and I will need to keep this workaround in mind if using ffmpeg scripts to create Ut Video proxies.

Scenario: Using ffmpeg to create video from an image file

This doesn’t work:
ffmpeg -loop 1 -t 5 -r 29.97 -s 1920x1080 -i r16g180b16-shotcut-rgb24.bmp -vf zscale=matrix=709:range=limited,format=yuv444p -c:v utvideo -an limited.avi

This does work:
ffmpeg -loop 1 -t 5 -r 29.97 -s 1920x1080 -i r16g180b16-shotcut-rgb24.bmp -vf zscale=matrix=709:range=limited,format=yuv444p -colorspace 1 -color_primaries 1 -color_trc 1 -c:v utvideo -an limited.avi

The problem:
This is the same issue as above. The filter graph is in BT.709 mode, but ffmpeg isn’t communicating that fact to the Ut Video codec. So again, Ut Video defaults itself to BT.601 colorspace which is confirmed by MediaInfo. Explicitly declaring the colorspace (even if using “1” for pass-thru) solves this problem.

I think I have enough test cases (around 40) to submit a bug report to ffmpeg. If anyone else could confirm or refute this hypothesis before I send it in, that would be awesome. In particular, my tests are very incomplete when the source material actually is in BT.601 colorspace. It would be great if one conversion script could handle a source in any colorspace or pixel format (bt601, bt709, RGB, RGBA, etc).

So, figuring out this madness is how I burned my afternoon yesterday. How about you guys? This Shotcut forum is a thing of beauty. A conversation with @D_S got me aware of Ut Video’s existence in the first place and it quickly replaced my H.264 CRF 12 proxies. Then @chris319 found a color problem that would have compromised my hopes of color grading on the proxies. Now it’s fixed, and on the audio side @shotcut informed me that AC-3 doesn’t have the delay issues that AAC does. @DRM got me thinking about proxy resolution again, and as I’ll describe on the proxy thread soon, I’m moving away from 480x270 because 270 isn’t divisible by 4 (long story). After pooling it all together, this new proxy setup is an editing dreamland and I doubt I would have found it on my own. Thanks everyone for the ideas!

shotcut · November 28, 2018, 6:55pm

Yes

shotcut · November 28, 2018, 7:01pm

It depends on “mlt_image_format” in the Other tab (consumer property in MLT terminology) and if certain filters or transitions are added that must convert to YUV for their job (a minority). One could also try using mlt_image_format=rgb24 or rgb24a with the Ut Video presets to workaround this problem.

Austin · November 28, 2018, 7:10pm

Yes, I considered that, but then the most technically-correct solution would be to create two Ut Video presets, one for YUV422 and one for RGB. I had not seen two presets created for other codecs, so I went with the one that is likely to get used the most. If you’re open to adding a RGB preset, I’ll research the lines that need to be added.

chris319 · November 29, 2018, 1:07am

Good work, Austin.

Austin · December 2, 2018, 12:20am

For anyone noticing color problems with ffmpeg scripts, I’ve now noticed shifts in ffmpeg 4.0.2 with codecs other than Ut Video when using zscale to change a JPEG-range video to MPEG-range. Even libx264 shifted colors in one scenario, but was fixed with the same “-colorspace 1 -color_primaries 1 -color_trc 1” hack I did for Ut Video. I’m thinking of leaving those three switches in there all the time just to be safe. Something to be aware of, as this issue may be more related to zscale output than to specific codecs.

chris319 · December 2, 2018, 3:26am

What is the difference between scale and zscale? The docs are not very clear on this.

Austin · December 2, 2018, 6:00am

The zscale filter references an external library called z.img that has only recently been compiled into ffmpeg as standard. Meanwhile, scale is internal to ffmpeg and has been around forever.

A seemingly large number of people online prefer zscale over scale because it does a better job with colors if any RGB-YUV or 601-709 conversions take place. I haven’t used it enough to verify this claim myself, but what I like about it so far is the addition of dithering options during a downsample. This helps reduce banding when making small proxies from 4K sources.

I upgraded to ffmpeg 4.1 and still having the problem. Same hack still fixes it.

chris319 · December 2, 2018, 9:33pm

Thanks for the explanation.