Does more disk space count as a suggestion? an 8tb Mybook(which has a standard 3.5 inch wd helioseal drive in it) is about 150 and would give you more than enough room to work with. https://amzn.to/2rhRQHf
What exactly are your âarchiveâ goals? I prefer to edit with a lossless intra frame codec like dnxhd. But I archive my final videos using H.264.
Your original DV video only has a chroma format of 4:1:1. So if you archive it as upsampled dnxhd, you are storing a lot of redundant information.
My advice would be to edit your files in the highest quality that you and muster, but then export them using H.264 4:2:0 with 60 frame GOP and 2 bframes. For 720p video, 20Mbps would be perceptually lossless given your source quality. And you wouldnât be losing any chroma resolution because the source was 4:1:1.
@D_S, a good idea. I could bump my secondary 1/2 TB drive to 4 TB for under $100 usd. But grandpaâs âtoysâ account has run pretty lean in this Christmas season.
@Brian - Archive plan? Of course I have an archive plan. Itâs just that it keeps changing 
. The initial strategy was to make a DVD of each grandchild over the last 21 years. I have a paper index of each tape so I can find the segments I need which span many of the 17 tape/captures but it means either transcoding all the 2 hour captures ahead of time, or transcoding the segments as I need them.
I hadnât decided the final product. Initially, I was going to make flash drives. But I have never seen gift sets of family flash drives. Despite the onslaught of history and technology there is nothing like a DVD or Blu-ray disk in a bookshelf display case. Flash drives tend to get put in a drawer and get lost. A suitable tag on them is several times the size of the flashdrive. Ugh!
I do thank you for revealing your process. I am taking careful notes since I donât want to reinvent that wheel all over again.
The conventional wisdom seems to be that we archive ALL family footage. Someday, somebody might want to do something with it. Itâs obvious though that it canât be archived in an intermediate form because it is too large. And not THAT valuableâŚ
-=Ken=-
I would never trust a DVD or a thumb drive for data archival, but M-Disc does look promising.
I see. You have two use cases going on here:
- Temporary storage of intermediate files so that you can sort through them all to use individualized clips of each recipient.
- A final export format and storage media to be distributed to the recipients
For the first use case, your goal would be to maximize quality, maintain edit friendliness and be able to fit all the files on the storage device you have available on your editing machine. For this, I could recommend intra-coded H.264. Just do some math to figure out what the highest bitrate you can use without filling up your hard drive. For 720p, I bet you could go as low as 50Mbps and keep perceptibly perfect intermediate files.
For the second use case, I would highly discourage any optical media. Your grandchildren will probably never own optical drives. I would encourage you to look at USB flash drive gift sets:
https://www.etsy.com/listing/619054093/wood-usb-box-with-personalization-usb
https://www.amazon.com/LONMAX-Wooden-Flash-Drive-Memroy/dp/B01N6JHTGC
https://www.amazon.com/Flash-Drive-Display-Memories-Bow-Tied/dp/B076BB11TY
For encoding, you would want a format that will be compatible with as many devices as possible. I would suggest MP4 file with H.264 60 frame GOP at 8Mbps. If you use the x264 encoder in 2 pass mode, the quality of 720p at 8Mbps will be astounding and it will play on any device for the foreseeable future.
Just to add to what @Hudson555x and @brian have said I work in IT(and have worked with archival data before) itâs been my experience that dvdâs lose data rather rapidly thanks to the organic dyes used, we had a large system with about 350 dvdâs and 10 years worth of data but only ~50% of that was accessible since they hadnât bothered to use any form of disc better than whatever was cheapest. You can get gold foil archival quality dvdâs but Iâm not sure Iâd trust those any better.
Regarding the flash drives depending on how many, and how often you make them it may be worthwhile to order blanks in bulk and get a CNC laser, making graphics is easy enough thanks to gimp/inkscape(and if you can handle what youâre doing in this thread I bet you can handle that. A small laser capable of engraving wood only costs 180 or so https://amzn.to/2Q8UTAF and at that point if you did just 10 flash drives youâd break even using a bulk pack like this https://amzn.to/2Q6jbeq past that youâre saving a ton.
I too had the cheapest CD\DVDâs backing up data over a decade ago, 2 discs out of 20 readable a few months ago.
One thing I do suggest going down the USB thumb drive route (while I donât trust them, I never had one fail) is sealing up in a water tight container, which would seal out the moisture and it would still be good if got lost 10 years from now because of the case. Or if they ammunition containers, make sure they put that inside their ammo box for secure storage.
Wow! Do you realize this forum remains one of the best on the Internet?
You folks here are down right amazing!
I missed seeing the flashdrive gift sets. What a novelty! Always something new under the sun. OK, that makes better sense. Iâll be checking that out real soon.
I guess I hadnât really said what my archive plan was. 17 2-hour Hi8 tapes scanned in as .DV files and 7 HDV files - all archived, for now, to a 2 TB Seagate GoFlex Home network NAS and a 1/2 TB WD Elements USB3 which is headed for the safe deposit box. In addition I have a 1 TB WD Elements USB3 local/working backup drive. So, three backups of the originals .DV files.
This is a hobby. Finishing the âpresentation flashdrivesâ project could take a couple years. Thatâs why I was thinking of wholesale making all 30+ hours of intermediate files until I knew I was really done. I guess I figured I could archive ISO files and burn new DVDs as I needed to in the future. I think you kind folks have disabused me of that notion.
Bravo - denizens of the Shotcut Forum - you have done it again!
Thank you,
-=Ken=-
@kenj69 Keep in mind spinning discs donât like bumps so if youâre going to keep one in a safe deposit box look towards the few that are shock mounted
@Hudson555x I actually have had USB flash drives fail, it wasnât moisture though(as long as they arenât connected while wet water is a non issue Iâve washed a few) but Heat causes random bit flips over time as Iâve more than tested with a flash drive attached to my car stereo slowly corrupting the mp3 files on hot summer days. If you do plan on using them for long term storage I suggest purchasing them from a T1 or T2 vendor (sandisk, kingston, mushkin) and skipping the generics which are that much easier to cook
I have finally found the proper filter settings for FFmpeg to match those I found in Avidemux. Someone in the Avidemux forum kindly showed me where they are âhidden.â Believe it or not, they are under Video Filters 10.140 named âpp.â They are called postprocessing subfilters. The command that I was looking for will appear like this:
-vf cubicipoldeint|autoq - or in abbreviated form -vf pp=ci|a [that is c, i, vertical brace and a]
http://ffmpeg.org/ffmpeg-filters.html#pp
So, a more complete command line would be:
ffmpeg -i in.dv -c:v dnxhd -vf âpp=ci|a,scale=1280x720,format=yuv422pâ -b:v 110M out.mov
For the rest of my footage the bwdif filter works great!
Concerning flash drive gift sets: A local store here sells little wooden boxes of various sizes for arts/crafts and so on. It seems to me that I can buy a premium flash drive and apply some sort of sticker on the box and drive - better quality, lower cost, less hassle. Glad to know about this!
-=Ken=-
@Hudson555x, @D_S it should not alarm you, but I participate in other forums than this one. I just had a discussion in Reddit/r/videography subreddit with someone who seems to have considerable experience with optical media. He makes compelling claims that optical media IS viable for today and into the foreseeable future and IS reliable for archiving family memories!
Now, my intention is not âyou and he go fight,â but I really donât know what to make of these conflicting claims. I used to work in IT as well and tend to trust you guys. Could you guys please check this out? Thanks -=Ken=-
Never worked in IT, and my very limited knowledge come from Google/YouTube (not the best idea I know). After experiencing data loss from CD/DVDâs. I just donât trust it, and will never use that form of media again. Just my own very personal opinion based on actual data loss. Iâm just the end user, the guy that lives down the block who doesnât mow the lawn on time with the rest of the neighbors, lol. I do like the concept of M-Disc, and will look forward to learning more about itâs development.
I donât know which media will last longer. But I do predict that flash drives will be more accessible in the future. New laptops donât even come with optical drives any more. And more and more young people are ditching redbox for streaming services. I predict that in 5 years more people will be able to read flash drives than optical drives. In fact, that is possibly true already today.
I do agree that in the distant future you will be able to find someone who can read the data off of an optical disc. Just like you can pay people to read data off of floppy disks today. I donât consider that very accessible.
@Hudson555x Iâm the one who works in IT, Iâm an enterprise IT admin and a product reviewer on the side
@kenj69 Heâs specifically stating that heâs using archival quality discs which are designed to last longer. Gold foil wonât oxidize the way silver foil does however they cost substantially more. They also still use an organic dye thatâs light and temperature sensitve so if theyâre not properly stored theyâre still liable to break down. He is correct you can physically damage a flash drive although iâve seen plenty of scratched and snapped dvdâs too. If heâs seeing the plugs fall to bits heâs not getting very good drives.
https://amzn.to/2PzRLbJ 13.48 for a 50 pack of standard discs
https://amzn.to/2rBQrf2 61.44 for a 50 pack of archival discs(which are cool because gold though)
I guess I should cap this discussion by thanking everyone who has contributed here. Even though I frequent other forums I can always count on a lively discussion on this one. BTW, try getting help for Kdenlive on the web. When I asked Kdenlive questions on the KDE forum it was not unusual to wait four days for single reply! This forum is a great resource.
After considerable experimentation I decided to I will use bwdif for deinterlacing and the nnedi filter for 11 minutes of problematic video. Here is my current configuration:
FFmpeg -y -hide_banner -loglevel warning -ss hh:mm:ss -i inputfile -t hh:mm:ss -vf "'NNedi=nnedi3_weights.bin':deint=all:field=a:qual=slow, '_denoizer_', scale=hd720:flags=spline, colormatrix=bt601:bt709, eq=saturation=1.3:contrast=1.2" -c:v libx264 -preset veryslow -crf 18 outputfile
Note: I havenât decided which denoiser filter to use yet.
Hope this helps someone.
Addendum:
I tried formatting a 32 GB Sandisk SD card so it would work on an HDTV, Blu-ray player or Android TV box. These devices are requiring FAT32 format. However, there are several varieties of FAT32 including vFAT and exFAT. As it turns out, only exFAT supports files more than 4 GB in size. So, I formatted the SD card with exFAT and copied a video file larger than 4 GB on it. Neither the TV nor the rather new Samsung Blu-ray player would recognize the exFAT formatted SD card!
So, I have given up. The current technology does not support what we are trying to do, at least not easily. And few care about the obstacles except for some of us technical types. Therefore, since the Linux world never developed a viable Blu-ray authoring application I will produce a few family videos using Linux DVD authoring software. I will backup/archive the ISO files of each DVD and, along with the raw captured family video I have and stash it in the safe deposit box. Someday, maybe, some family member may want to do something with this old video. They will just have to figure out how.
Blessings,
-=Ken=-
exFAT and similar support varries based on age, some newer HDTV and players do support it but itâs not universal(and upgrading hardware just for that is silly) NTFS support might exist(once again itâs per the hardware maker to support that some d some donât) or you can split your file into multiple segments(which a 720hd file would have to be rather long to be larger than 4gb) and include a playlist. All that said, as long as you get good dvdâs it seems you have a good solution for now, if you give them the iso files those are natively mountable on most modern operating systems as well.
SD Cards come in different speeds and classes. Not all devices support all classes or speeds.
https://www.sdcard.org/developers/overview/speed_class/
And they have their own format program for Windows/Mac, and many companies and tech people all recommend this format program for SD cards.
Thanks for the info on the SD cards!
Mine are 32 GB SanDisk Ultra @10 class rated for 80 MB/s read speed.
-=Ken=-
Brian, I hope you are still around. I have been reading up on x264 encoding but I still donât understand it very well.
I encoded my first 7 minute segment of poor video using this encoding string:
-i input.dv -vf ânnedi=ânnedi3_weights.binâ:deint=all:field=a:qual=slow, hqdn3d=10:10:40:13, scale=hd720:flags=spline, colormatrix=bt601:bt709, eq=saturation=1.3:contrast=1.2â -c:v libx264 -preset veryslow -crf 18 output.mov
When I check with MediaInfo it says the Overall bit rate is 20.3 Mb/s. Is that good enough for âintra-codecâ use or do I need to do further digging to get this right?
-=Ken=-
In order to have intra only encoding, you need to limit the GOP size to 1 so that it only creates I frames. Try adding â-intraâ: