I see it is using Metal, but maybe that does not work good or work good on non-Apple Silicon. You can run Shotcut’s whisper directly like /Applications/Shotcut.app/Contents/MacOS/whisper-cli with different args to see if you get a better response. What different options? Well, I am not sure; I am not a whisper expert. Upon reviewing them and based on a Web search it seems it may not be possible without a recompile!
I just made a quick test on my M1 with part of an episode of The King of Queens, and it did a great job.
Can you configure Shotcut to use the homebrew whisper and test it within Shotcut?
Update: I just tested it on my Intel Mac, and it was horribly slow and wrong as you experienced. We have optimized for Apple Silicon, and that is unlikely to be changed.