A known bug already reported between FFmpeg and videotoolbox in macOS. Workaround is to use HEVC or turn off hardware encoder.