Is it possible in MLT-Framework to refer Video-Files inside ZIP/7Zip-Archives?

Hi,

I was wondering if it is possible to use a video from inside a Zip/7Zip-Archive to save space on my disk. Can someone please clarify on this and if this is not possible maybe can this be an addition to the project?

For clarification I want to share this StackOverflow-Link here: C# Access text file in zip archive - Stack Overflow
(its about a text file, but I think you will get it)

This would be very convinient but I can also think of the case that Shotcut/Melt doesn’t support the handling of Archive-Files for File-Retrieval.

What if exposing the contents of a zip archive was first done by the operating system so that the zip file is transparent to Shotcut? There are tools that can mount an archive to a mount point (Linux) or a drive letter (Windows):

Linux:

archivemount
https://www.cybernoia.de/software/archivemount.html
https://github.com/cybernoid/archivemount

fuse-zip
https://bitbucket.org/agalanin/fuse-zip/

Windows:

Pismo File Mount
https://pismotec.com/pfm/

WinArchiver
https://www.winarchiver.com/index.htm

But what if I have multiple Zip-Archives and I need to switch fast between them or need them for multiple Projects?

That would mean that I need to mount every ZIP-Archive I want to use. Thats a huge overhead instead of just providing a Archive-internal relative File path which is then opened by Shotcut to just retrieve the files that are needed to use.

Reading a zip archive even for a single file is massive overhead. There is nothing fast about this process. :slight_smile: Real-time video editing would probably not be possible in this format because the reading and decoding phase has too much latency.

I tried compressing an MP4 H.264 file with Zip just out of curiosity. As expected, the compression ratio was less than 10%. The compression by H.264 is already pretty extreme. Are you using a video format that gets more than 10% compression ratio from Zip? If not, is Zip really worth it?

I am mostly talking about the Framework and Melt as is. This is just for clarification :slight_smile:

I am using OBS as I am a streamer an saving my VODs as KMV-Files so nearly compression-free. These files can be really big (~20-30 GB for 6-9 Hours of material) and as I am streaming nearly every two days I have a lot of these. And I have created a Rendering Management Tool to create MLT-Files to split the big Video-Files into smaller Chunks for Rendering and then direct Melt to actually render the Outputs.

And the Rendering takes time and is costly. So I would like to save some disk space until I have processed a big video file and my thought was to use Archives for the remaining big files. I could in fact load the Video File by myself in my Code and save in on disk first to use it for Melt. But I also thought that this could be a cool function for Melt.

MKV is the container… which video codec is being used?

I ask because depending on the answer, it might be possible to get the benefits you want without the overhead and incompatibility of a zip file.

As I’m sure you already know, zip compression is lossless. Compressed files can be perfectly reconstructed. In the video world, this roughly equates to the lossless video codecs.

So my curiosity is whether converting your videos to lossless Ut Video format would provide compression ratios and speeds that were on par with a Zip file. After all, Ut Video and Zip formats are both based on Huffman encoding. I would expect close-enough file sizes due to their common ancestry. (This is of course assuming your source video format has basically no compression. Zip format may show a slight lead because it isn’t limited to frame boundaries like Ut Video is. But that lead may be insignificant.)

If this turns out to be true, then the Zip format could be entirely skipped, and you could go direct to Ut Video, which is a true first-class video format that is immediately usable by Shotcut. This would also give you the ability to preview videos in VLC without having to unpack them, which could be great for locating the archived footage you need in a project.

Batch conversion to Ut Video can be done with ffmpeg, which in essence would replace the pkzip command:

ffmpeg -i MassiveStreamedVideoFile.mkv -c:v utvideo -pred median -c:a copy HopefullyLessMassiveVideoFile.mkv

The -pred median part can provide some increased compression efficiency at the slight cost of speed. The default is -pred left which is really fast, perhaps as fast as pkzip.

Also, there’s the option to change the pixel format if desired by adding one of these flags before the -c:v part:

-pix_fmt yuv444p
-pix_fmt yuv422p
-pix_fmt yuv420p

These are ordered from largest file size to lowest file size. If your source files happen to be RGB or YUV444, you could save even more space here by converting to YUV422. The quality difference would not be noticed since Shotcut edits in YUV422 anyway (assuming you aren’t overriding it to operate in RGB mode, which would provide no benefit since YouTube will pull RGB down to YUV420 anyway).

For even more compression, there is the option of using libx264 in lossless mode with a GOP of 30 or higher. The -g parameter sets the GOP. For streaming video games, 60 or higher may be ideal. libx264 might (or might not) take slightly longer to convert, but the space savings could be drastic:

ffmpeg -i MassiveStreamedVideoFile.mkv -c:v libx264 -qp 0 -g 60 -b 0 -preset ultrafast -movflags +faststart -c:a copy StunninglyTinyVideoFile.mkv

The -pix_fmt flags apply here too, if the source was RGB/YUV444 and you want to drop to YUV422 or YUV420.

The whole “can find stuff first with VLC” benefit applies to libx264 as well. Actually, I’d recommend trying the libx264 option first over Ut Video, now that I think about the space savings. I’m just too lazy to rewrite everything I already wrote. :grin:

I’ve gone down the lossless video rabbit hole because a Zip file provides lossless quality, and I wanted to provide an equivalent solution. But if you don’t actually need lossless and are content with “good enough”, then libx264 can go really fast with huge space savings if allowed to work in veryfast mode:

ffmpeg -i MassiveStreamedVideoFile.mkv -c:v libx264 -crf 16 -g 240 -b 0 -preset veryfast -movflags +faststart -c:a copy OhWowItsSoTiny.mkv

1 Like

Whoa, thats a huge load of information. Thanks for this first.

Then I am actually not that deeply into codecs and no video guy at all. All I got from OBS as information is that I am using the NVENC Codec for video and NV12 as the color scheme. As you already said, maybe this is the best option to have less bigger files in the beginning.

Can you give an estimate on how big the weight loss would be by using the changes you have suggested?

The thing with ffmpeg and the converting seems to be also sufficient but I need to check and validate this.

Okay, so but it seems that ZIP is really not the best solution for this.

Hmmmm, so, that information changed the entire conversation. :smile:

20-30 GB per 6-9 hours averages to 7400 kbps data rate. NVENC means the codec is probably either H.264 or H.265, and NV12 means color is 4:2:0.

These video files are already super-compressed. You’re now in the same boat I was in at the beginning, which is less than 10% space reduction by using Zip over H.264.

If NVENC is coding to H.264, there is potential to reduce the data rate by using H.265 instead.

Aside from that, it would take a non-trivial amount of computer time to make these files smaller in a meaningful way, and there would be a visual quality penalty due to the conversion to a new format.

Basically, the files are what they are. If the workflow can’t change, then it’s time to buy more hard drives.


EDIT: Well, I’ll offer a variant and let you decide if it’s worth your time…

NVENC prior to 20xx/30xx sometimes made files that were up to 3x larger than software encoding for similar quality. You could try this command to see if file size comes down without a noticeable impact on visual quality:

ffmpeg -i MassiveStreamedVideoFile.mkv -c:v libx264 -crf 18 -g 240 -b 0 -preset veryfast -movflags +faststart -c:a copy GraspingAtStraws.mkv

If you are happy with the quality and the file size is lower, then that’s the ticket. But if conversion takes too long to be worth it, then you’re probably stuck with the original files just the way they are.

1 Like

Okay thanks either way. I will still try your solutions from above. Thanks man :slight_smile:

1 Like

I just came here to say that Windows has a feature that you can enable compression on a specific folder without making a zip archive. This would work with Shotcut because it is handled by the operating system and Shotcut does not know about it.

https://windows.tips.net/T005671_Understanding_Compressed_Folders.html

However, I agree with @Austin 's comments: this is unlikely to produce satisfactory results since it will make Shotcut feel slower and the files are most likely to already be compressed to a high degree.