What is your Shotcut version (see Help > About Shotcut)? Is it 32-bit?
20.06.28 x64 portable
Can you repeat the problem? If so, what are the steps?
H.264 lossless export using h264_nvenc is lossly. Worked in previous versions. For example, h.264 lossless using libx264 had a size of 314 MB. Same export using h264_nvenc had a size of 22 MB. Tested the same in 20.04.12 x64 portable and h264_nvenc. had a size of 115 MB.
Export. Check Use Hardware Encoder. Choose H.264 Lossless. Change export resolution. Use Lanczos interpolation. Export.
Unfortunately, the NVENC from the previous version was not actually lossless. It is virtually impossible for a hardware encoder to produce a smaller lossless file than libx264, and definitely not a lossless file in half the size. The NVENC encode from the previous Shotcut version may have been at very high quality, but definitely not lossless. As Dan hinted, very few hardware encoders support lossless mode.
The best solution is to run the 20.04.12 NVENC encode again, but check the log when it’s done to figure out which settings actually got used. Then replicate those settings on 20.06.28. It seems that some setting in 20.06.28 is defaulting to a different (lesser quality) value than it did before. It may be necessary to build a custom export preset to make hardware exporting convenient again.
EDIT: Sorry, I didn’t notice that you included logs before. Awesome! I took a look, and unfortunately, the NVENC log doesn’t list the settings it used like newer cards and drivers.
However, I did notice a major change elsewhere… In Shotcut 20.04.12, the H.264 Lossless export preset has a GOP of 1 (all-intra). In Shotcut 20.06.28, the GOP is 25 (no longer all-intra). This could have a huge impact on compression ratio depending on the nature of the video.
Basically, file size can’t be the indicator of quality in this comparison due to the GOP change. Since true lossless hardware encoding isn’t an option with that model of GTX, you may have to eyeball the next best settings until you’re satisfied with the results.
The Nvidia support matrix kinda suggests the GTX 650 doesn’t support lossless: