I did not find anything like a bad default max bitrate.
Related
52 seconds with cq10, 80% and cq26, 50%
The above link threshold is 24, but your footage might be different if it is more challenging. In other words, you did not try a low enough quality. You are comparing high with higher, and NVENC is not giving you higher because it has some internal limiter. Maybe it worked for you with H.264 because it is less efficient.