There is a known bug with the H.264 hardware encoder on some Macs or macOS versions. Either use HEVC with hardware encoder or turn off hardware encoder. The software H.264 encoder is efficient enough that there is really no need to use hardware acceleration.