Intermediate Files for Editing

Help/How To
26

Sounds like two things in play here. One is RGB vs sRGB to YUV colorspace conversion. The other is MPEG range vs JPEG range video. You’re probably familiar with both issues, but I’ll give a brief explanation for everyone else.

MPEG range is more of a broadcast restriction that says RGB values have to be between 16 and 235. 16 is black and all-235s is white. JPEG range is 0-255. This can be controlled with the ffmpeg -color_range parameter. Converting improperly between them can cause raised or crushed whites/blacks plus color shifts in-between.

The other issue is simple colorspace conversion error. Maybe converting from PNG instead of BMP can be a workaround.

I’ve noticed color shifts exporting from Shotcut that I don’t get using ffmpeg directly. Not sure the issue there either. Will write more if I get a chance to test.

27

It’s not a bmp issue. The same bmp gives good colors when converted using ffmpeg directly and not going through ShotCut.

I think this is it. I’m not worried about it, though as I can use my direct ffmpeg scripts.

28

Forgot to follow up on this… In theory, x264 at CRF 0 is true lossless. The rounding error is probably colorspace conversions and there’s not a lot that can be done about that.

29

There will always be rounding error in calculations involving YUV due to the use of floating-point coefficients. There is YCoCg but it doesn’t subsample well. I don’t think you could do 4:2:2 or 4:2:0 in YCoCg.

I was going to use Huffyuv as my interpositive format until I discovered these color errors, so I’m going with good ol’ x.264 with CRF 0. x.264 is a real workhorse.

I am borrowing an old film term:

30

Interesting note, in further research I discovered WHY I wasn’t seeing a color shift on my end is that my camera is recording as YUV by default according to mediainfo, interesting discovery to say the least I wonder if it has an option to go between that and rgb internally.

31

I lost a little in translation… were you seeing color shifts with command-line ffmpeg as well, or only Shotcut presets?

I just did a round of tests using the same color you did using ffmpeg:

ffmpeg -f image2 -s 720x480 -r 29.97 -i "016,180,016.png" -pix_fmt yuv420p -color_range mpeg -c:v ffv1 yuv420p-mpeg.ffv1.avi

I changed -c:v to be HuffYUV, UtVideo, FFV1, and ProRes (and yes some complained that yuv420p was incompatible). I played back their outputted videos with MPC-BE media player on Windows 10, did a screen capture, and eye-dropped the frame to get the RGB. On every codec, the RGB values were within 1. This was true for -color_range of JPEG or MPEG.

I got confused because you ended up with x264 CRF 0. I thought the Shotcut color shifts didn’t bother you because you used scripts, but if you used scripts, then HuffYUV would still be a candidate and you wouldn’t need x264 CRF 0 … so I got confused.

If your ffmpeg is shifting colors, two thoughts come to mind. One is that your ffmpeg is fine and it’s a media player that doesn’t know how to do MPEG/JPEG color range conversion correctly. MPC-BE does. The second option could be an old ffmpeg. I’m using 4.0.2 and get no shifts with the lossless codecs.

Having said that, you’ll probably get better compression with x264 CRF 0 so maybe that’s still a good deal. :slight_smile:

32

If you checked the stack exchange link I posted part of why he’s seeing shifts and you’re not is the usage of a BMP vs PNG. Unless they’ve updated it(they haven’t as far as i’m aware) the BMP decoder in ffmpeg is an antique (2005) and requires manually declaring what it’s color space is.

33

I assume the article is old. I did tests with both PNG and BMP, and the results were identical and unshifted. @chris319 also stated that it was not a BMP issue. Colorspace shouldn’t have to be declared because all these formats with the exception of ProRes have native rgb24 support, same as BMP, so it should just work.

34

It is indeed old although I can’t find any evidence that ffmpeg has been updated to correct it, shouldn’t but it very well may need it.

35

Agreed. It makes me appreciate x.264 more.

The problem is, I’m still seeing color shifts in ShotCut’s H.264 presets. I’ll test some more to make sure.

36

Notwithstanding the subsequent discussion into how each intermediate files are working in practise, my takeaway from the thread is:

  1. Try using the original files. If they work in the editor without stuttering etc (because the computer is struggling to decode the frame level information on the fly) then all good. No further action - work with the originals.

  2. If the computer does struggle (stuttering etc), then export to ProRes at 100% quality.

NB: Whilst the intermediate creates a full frame for every frame (in theory improving the editing experience) it does also create a larger file and may therefore cause the editor to suffer similar problems to the original files, albeit for a different reason.

(As an aside… Where does Cineform fit into the discussion? Go Pro’s editor converts the source MP4 files into Cineform to edit. No real choice for the user… Why might Go Pro avoid allowing the user of their software to use the original file?)

37

Sounds like you nailed it. I might add two very minor additions:

1b) If the original file is variable frame rate like a cell phone, always create an intermediate even if the editor can play the original back. The playback will not be smooth or sync well until it is constant frame rate.

2b) If the computer still struggles even with ProRes, there is the option of going to proxies and using the original file for the final render (no need to keep the ProRes first attempt).

Cineform is designed to be an intermediate format like ProRes and DNxHD. It is quite capable with multi-generation encoding with no loss, and it even threads well. The main reason we haven’t mentioned it is that, to my knowledge, ffmpeg does not have an encoder for it yet. Since Shotcut is built on top of ffmpeg, that means you couldn’t export to it either.

GoPro paid big bucks to buy the Cineform format. They probably convert everything in Studio because users would complain if native H.264 stuttered. GoPro Tech Support doesn’t want to ask people how fast their computers are, so they chose a format that will work on pretty much all computers. Their audience is average users and kids, not video professionals.

38

Looks like it is in ffmpeg but it’s still a work in progress, gopro open sourced it in 2017
https://gopro.com/news/gopro-open-sources-the-cineform-codec
https://wiki.multimedia.cx/index.php/Cineform_RAW

39

To make the engine easier to implement. Seeking is as simple as going to the frame in one format and codec, and decoding reference frames is not required. Of course, it is also faster to seek this way. Maybe everything also gets conformed to the same resolution and frame rate in the process, which can also make it simpler.

All this discussion about RGB values is very short on the description of the process and as Austin mentioned color spaces and ranges. Please be aware that by default MLT renders everything to YUV 4:2:2 with MPEG ranges. If you are working with a full range source or targeting a RGB codec, you should add mlt_image_format=rgb24 to the Other tab.

40

Now I am getting some grievous color errors in Shotcut no matter which codec I use. Same methodology as yesterday but different results.

The mp4 files I coded with ffmpeg on the command line are still good. Weird. I tried different settings in Shotcut with no improvement.

I can’t see the commands Shotcut is sending to ffmpeg so ???

The one change since yesterday is that I ran DisplayCal with a new setting and reloaded the table it generates. I hate to think this is the problem, maybe related to my GPU?

41

Shotcut is not a ffmpeg command line frontend (except for Properties > Convert and More Information).

42

I’m thinking I need to disable GPU processing but I am not saving the project as an MLT file. I am loading the bmp directly into shotcut and exporting it. I will see what’s the deal with an MLT file.

Just checked. Settings -> GPU Effects is unchecked.

43

I did some testing and found RGB output (mlt_image_format=rgb24 and PNG or HuffYUV with pix_fmt=rgb24) from Shotcut is good but anything using YUV is darkened. I will work on a fix.

Testing Details

To assess the color handling, first start with RGB handling input-through-output along with a test process. I plan to use ffmpeg to convert one frame of something to PNG, and open it in paint.exe to use its color picker and edit colors tools to get the resulting RGB values. (You can use another tool such as Gimp, but I did not want to wait for it to load). I am using RGB (16,180,16) = #10B410 (#FF10B410) in Shotcut’s color generator). I am using Shotcut 18.10.08 and ffmpeg 4.0.x that comes with it.

RGB24 -> RGB24 (full range)

Shotcut: Using the color generator and the PNG export preset with mlt_image_format=rgb24, I get
(16,180,16).

Using ffmpeg -f lavfi -i color=#10B410 -t 0.04 -pix_fmt rgb24 /i/testing/r16g180b16-ffmpeg-rgb24.png I get (15,177,14) !! That’s not good! Let’s try using Shotcut’s PNG as input:

Using ffmpeg -i '/i/testing/r16g180b16-shotcut-rgb24-00001.png' -pix_fmt rgb24 /i/testing/r16g180b16-ffmpeg-rgb24.png I get (16,180,16).

OK, now we have a combination of things that we can trust.

x264 yuv444p

Using ffmpeg

ffmpeg -i '/i/testing/r16g180b16-shotcut-rgb24-00001.png' -pix_fmt yuv444p /i/testing/r16g180b16-ffmpeg-yuv444p.mp4
ffmpeg -i '/i/testing/r16g180b16-ffmpeg-yuv444p.mp4' -pix_fmt rgb24 /i/testing/r16g180b16-ffmpeg-yuv444p.png

I get (16,179,15).

Shotcut: Using the color generator and the export defaults with other:

mlt_image_format=rgb24
pix_fmt=yuv444p

and ffmpeg -i '/i/testing/r16g180b16-shotcut-yuv444p.mp4' -pix_fmt rgb24 /i/testing/r16g180b16-shotcut-yuv444p.png
I get (4,154,10). That is a problem.

x264 yuvj444p (full range)

ffmpeg -i '/i/testing/r16g180b16-shotcut-rgb24-00001.png' -pix_fmt yuvj444p /i/testing/r16g180b16-ffmpeg-yuvj444p.mp4
ffmpeg -i '/i/testing/r16g180b16-ffmpeg-yuvj444p.mp4' -pix_fmt rgb24 /i/testing/r16g180b16-ffmpeg-yuvj444p.png

I get (15,180,16).

Shotcut: Using the color generator and the export defaults with other:

mlt_image_format=rgb24
pix_fmt=yuvj444p

and ffmpeg -i '/i/testing/r16g180b16-shotcut-yuv444p.mp4' -pix_fmt rgb24 /i/testing/r16g180b16-shotcut-yuv444p.png
I get (3,154,12). That is a problem.

x264 yuv422p

ffmpeg -i '/i/testing/r16g180b16-shotcut-rgb24-00001.png' -pix_fmt yuv422p /i/testing/r16g180b16-ffmpeg-yuv422p.mp4
ffmpeg -i '/i/testing/r16g180b16-ffmpeg-yuv422p.mp4' -pix_fmt rgb24 /i/testing/r16g180b16-ffmpeg-yuv422p.png

I get (15,177,14).

Shotcut: Using the color generator and the export defaults with other:

pix_fmt=yuv422p

and ffmpeg -i '/i/testing/r16g180b16-shotcut-yuv422p.mp4' -pix_fmt rgb24 /i/testing/r16g180b16-shotcut-yuv422p.png
I get (1,153,7). That is a problem.

HuffYUV

I get similar poor results from Shotcut using huffyuv yuv422p but not with

mlt_image_format=rgb24
pix_fmt=rgb24
44

Why are you testing with PNG and not BMP?

45

Arbitrary choice.
I found a flaw with my test procedure. The Shotcut/MLT color generator tries to generate in YUV if the image was requested in that image format, not RGB. It uses some suspicious coefficients for that. If I use the PNG source that I used for ffmpeg in Shotcut, then I get (13,178,13) for x264 yuv422p, which is much closer to expected and ffmpeg’s result.