Technical Discussion: Interlacing Revisited

Does more disk space count as a suggestion? an 8tb Mybook(which has a standard 3.5 inch wd helioseal drive in it) is about 150 and would give you more than enough room to work with. https://amzn.to/2rhRQHf

What exactly are your “archive” goals? I prefer to edit with a lossless intra frame codec like dnxhd. But I archive my final videos using H.264.

Your original DV video only has a chroma format of 4:1:1. So if you archive it as upsampled dnxhd, you are storing a lot of redundant information.

My advice would be to edit your files in the highest quality that you and muster, but then export them using H.264 4:2:0 with 60 frame GOP and 2 bframes. For 720p video, 20Mbps would be perceptually lossless given your source quality. And you wouldn’t be losing any chroma resolution because the source was 4:1:1.

@D_S, a good idea. I could bump my secondary 1/2 TB drive to 4 TB for under $100 usd. But grandpa’s ‘toys’ account has run pretty lean in this Christmas season.

@Brian - Archive plan? Of course I have an archive plan. It’s just that it keeps changing :laughing:. The initial strategy was to make a DVD of each grandchild over the last 21 years. I have a paper index of each tape so I can find the segments I need which span many of the 17 tape/captures but it means either transcoding all the 2 hour captures ahead of time, or transcoding the segments as I need them.

I hadn’t decided the final product. Initially, I was going to make flash drives. But I have never seen gift sets of family flash drives. Despite the onslaught of history and technology there is nothing like a DVD or Blu-ray disk in a bookshelf display case. Flash drives tend to get put in a drawer and get lost. A suitable tag on them is several times the size of the flashdrive. Ugh!

I do thank you for revealing your process. I am taking careful notes since I don’t want to reinvent that wheel all over again.

The conventional wisdom seems to be that we archive ALL family footage. Someday, somebody might want to do something with it. It’s obvious though that it can’t be archived in an intermediate form because it is too large. And not THAT valuable…

-=Ken=-

I would never trust a DVD or a thumb drive for data archival, but M-Disc does look promising.

I see. You have two use cases going on here:

  1. Temporary storage of intermediate files so that you can sort through them all to use individualized clips of each recipient.
  2. A final export format and storage media to be distributed to the recipients

For the first use case, your goal would be to maximize quality, maintain edit friendliness and be able to fit all the files on the storage device you have available on your editing machine. For this, I could recommend intra-coded H.264. Just do some math to figure out what the highest bitrate you can use without filling up your hard drive. For 720p, I bet you could go as low as 50Mbps and keep perceptibly perfect intermediate files.

For the second use case, I would highly discourage any optical media. Your grandchildren will probably never own optical drives. I would encourage you to look at USB flash drive gift sets:

https://www.etsy.com/listing/619054093/wood-usb-box-with-personalization-usb

https://www.amazon.com/LONMAX-Wooden-Flash-Drive-Memroy/dp/B01N6JHTGC

https://www.amazon.com/Flash-Drive-Display-Memories-Bow-Tied/dp/B076BB11TY

For encoding, you would want a format that will be compatible with as many devices as possible. I would suggest MP4 file with H.264 60 frame GOP at 8Mbps. If you use the x264 encoder in 2 pass mode, the quality of 720p at 8Mbps will be astounding and it will play on any device for the foreseeable future.

Just to add to what @Hudson555x and @brian have said I work in IT(and have worked with archival data before) it’s been my experience that dvd’s lose data rather rapidly thanks to the organic dyes used, we had a large system with about 350 dvd’s and 10 years worth of data but only ~50% of that was accessible since they hadn’t bothered to use any form of disc better than whatever was cheapest. You can get gold foil archival quality dvd’s but I’m not sure I’d trust those any better.

Regarding the flash drives depending on how many, and how often you make them it may be worthwhile to order blanks in bulk and get a CNC laser, making graphics is easy enough thanks to gimp/inkscape(and if you can handle what you’re doing in this thread I bet you can handle that. A small laser capable of engraving wood only costs 180 or so https://amzn.to/2Q8UTAF and at that point if you did just 10 flash drives you’d break even using a bulk pack like this https://amzn.to/2Q6jbeq past that you’re saving a ton.

I too had the cheapest CD\DVD’s backing up data over a decade ago, 2 discs out of 20 readable a few months ago.

One thing I do suggest going down the USB thumb drive route (while I don’t trust them, I never had one fail) is sealing up in a water tight container, which would seal out the moisture and it would still be good if got lost 10 years from now because of the case. Or if they ammunition containers, make sure they put that inside their ammo box for secure storage.

Wow! Do you realize this forum remains one of the best on the Internet?
You folks here are down right amazing!

I missed seeing the flashdrive gift sets. What a novelty! Always something new under the sun. OK, that makes better sense. I’ll be checking that out real soon.

I guess I hadn’t really said what my archive plan was. 17 2-hour Hi8 tapes scanned in as .DV files and 7 HDV files - all archived, for now, to a 2 TB Seagate GoFlex Home network NAS and a 1/2 TB WD Elements USB3 which is headed for the safe deposit box. In addition I have a 1 TB WD Elements USB3 local/working backup drive. So, three backups of the originals .DV files.

This is a hobby. Finishing the “presentation flashdrives” project could take a couple years. That’s why I was thinking of wholesale making all 30+ hours of intermediate files until I knew I was really done. I guess I figured I could archive ISO files and burn new DVDs as I needed to in the future. I think you kind folks have disabused me of that notion.

Bravo - denizens of the Shotcut Forum - you have done it again!

Thank you,

-=Ken=-

@kenj69 Keep in mind spinning discs don’t like bumps so if you’re going to keep one in a safe deposit box look towards the few that are shock mounted

@Hudson555x I actually have had USB flash drives fail, it wasn’t moisture though(as long as they aren’t connected while wet water is a non issue I’ve washed a few) but Heat causes random bit flips over time as I’ve more than tested with a flash drive attached to my car stereo slowly corrupting the mp3 files on hot summer days. If you do plan on using them for long term storage I suggest purchasing them from a T1 or T2 vendor (sandisk, kingston, mushkin) and skipping the generics which are that much easier to cook

I have finally found the proper filter settings for FFmpeg to match those I found in Avidemux. Someone in the Avidemux forum kindly showed me where they are “hidden.” Believe it or not, they are under Video Filters 10.140 named “pp.” They are called postprocessing subfilters. The command that I was looking for will appear like this:

-vf cubicipoldeint|autoq - or in abbreviated form -vf pp=ci|a [that is c, i, vertical brace and a]

http://ffmpeg.org/ffmpeg-filters.html#pp

So, a more complete command line would be:

ffmpeg -i in.dv -c:v dnxhd -vf “pp=ci|a,scale=1280x720,format=yuv422p” -b:v 110M out.mov

For the rest of my footage the bwdif filter works great!

Concerning flash drive gift sets: A local store here sells little wooden boxes of various sizes for arts/crafts and so on. It seems to me that I can buy a premium flash drive and apply some sort of sticker on the box and drive - better quality, lower cost, less hassle. Glad to know about this!

-=Ken=-

@Hudson555x, @D_S it should not alarm you, but I participate in other forums than this one. I just had a discussion in Reddit/r/videography subreddit with someone who seems to have considerable experience with optical media. He makes compelling claims that optical media IS viable for today and into the foreseeable future and IS reliable for archiving family memories!

Now, my intention is not “you and he go fight,” but I really don’t know what to make of these conflicting claims. I used to work in IT as well and tend to trust you guys. Could you guys please check this out? Thanks -=Ken=-

Never worked in IT, and my very limited knowledge come from Google/YouTube (not the best idea I know). After experiencing data loss from CD/DVD’s. I just don’t trust it, and will never use that form of media again. Just my own very personal opinion based on actual data loss. I’m just the end user, the guy that lives down the block who doesn’t mow the lawn on time with the rest of the neighbors, lol. I do like the concept of M-Disc, and will look forward to learning more about it’s development.

I don’t know which media will last longer. But I do predict that flash drives will be more accessible in the future. New laptops don’t even come with optical drives any more. And more and more young people are ditching redbox for streaming services. I predict that in 5 years more people will be able to read flash drives than optical drives. In fact, that is possibly true already today.

I do agree that in the distant future you will be able to find someone who can read the data off of an optical disc. Just like you can pay people to read data off of floppy disks today. I don’t consider that very accessible.

@Hudson555x I’m the one who works in IT, I’m an enterprise IT admin and a product reviewer on the side

@kenj69 He’s specifically stating that he’s using archival quality discs which are designed to last longer. Gold foil won’t oxidize the way silver foil does however they cost substantially more. They also still use an organic dye that’s light and temperature sensitve so if they’re not properly stored they’re still liable to break down. He is correct you can physically damage a flash drive although i’ve seen plenty of scratched and snapped dvd’s too. If he’s seeing the plugs fall to bits he’s not getting very good drives.

https://amzn.to/2PzRLbJ 13.48 for a 50 pack of standard discs
https://amzn.to/2rBQrf2 61.44 for a 50 pack of archival discs(which are cool because gold though)

I guess I should cap this discussion by thanking everyone who has contributed here. Even though I frequent other forums I can always count on a lively discussion on this one. BTW, try getting help for Kdenlive on the web. When I asked Kdenlive questions on the KDE forum it was not unusual to wait four days for single reply! This forum is a great resource.

After considerable experimentation I decided to I will use bwdif for deinterlacing and the nnedi filter for 11 minutes of problematic video. Here is my current configuration:

FFmpeg -y -hide_banner -loglevel warning -ss hh:mm:ss -i inputfile -t hh:mm:ss -vf "'NNedi=nnedi3_weights.bin':deint=all:field=a:qual=slow, '_denoizer_', scale=hd720:flags=spline, colormatrix=bt601:bt709, eq=saturation=1.3:contrast=1.2" -c:v libx264 -preset veryslow -crf 18 outputfile

Note: I haven’t decided which denoiser filter to use yet.
Hope this helps someone.

Addendum:
I tried formatting a 32 GB Sandisk SD card so it would work on an HDTV, Blu-ray player or Android TV box. These devices are requiring FAT32 format. However, there are several varieties of FAT32 including vFAT and exFAT. As it turns out, only exFAT supports files more than 4 GB in size. So, I formatted the SD card with exFAT and copied a video file larger than 4 GB on it. Neither the TV nor the rather new Samsung Blu-ray player would recognize the exFAT formatted SD card!

So, I have given up. The current technology does not support what we are trying to do, at least not easily. And few care about the obstacles except for some of us technical types. Therefore, since the Linux world never developed a viable Blu-ray authoring application I will produce a few family videos using Linux DVD authoring software. I will backup/archive the ISO files of each DVD and, along with the raw captured family video I have and stash it in the safe deposit box. Someday, maybe, some family member may want to do something with this old video. They will just have to figure out how.

Blessings,

-=Ken=-

exFAT and similar support varries based on age, some newer HDTV and players do support it but it’s not universal(and upgrading hardware just for that is silly) NTFS support might exist(once again it’s per the hardware maker to support that some d some don’t) or you can split your file into multiple segments(which a 720hd file would have to be rather long to be larger than 4gb) and include a playlist. All that said, as long as you get good dvd’s it seems you have a good solution for now, if you give them the iso files those are natively mountable on most modern operating systems as well.

SD Cards come in different speeds and classes. Not all devices support all classes or speeds.
https://www.sdcard.org/developers/overview/speed_class/

And they have their own format program for Windows/Mac, and many companies and tech people all recommend this format program for SD cards.

Thanks for the info on the SD cards!
Mine are 32 GB SanDisk Ultra @10 class rated for 80 MB/s read speed.

-=Ken=-

Brian, I hope you are still around. I have been reading up on x264 encoding but I still don’t understand it very well.

I encoded my first 7 minute segment of poor video using this encoding string:

-i input.dv -vf “nnedi=‘nnedi3_weights.bin’:deint=all:field=a:qual=slow, hqdn3d=10:10:40:13, scale=hd720:flags=spline, colormatrix=bt601:bt709, eq=saturation=1.3:contrast=1.2” -c:v libx264 -preset veryslow -crf 18 output.mov

When I check with MediaInfo it says the Overall bit rate is 20.3 Mb/s. Is that good enough for ‘intra-codec’ use or do I need to do further digging to get this right?

-=Ken=-

In order to have intra only encoding, you need to limit the GOP size to 1 so that it only creates I frames. Try adding “-intra”: