No, VideoToolbox is a macOS framework that provides an abstraction over different hardware.
What about Windows nvenc for OSX?
NVIDIA says NVENC is for Windows and Linux only, and Apple and NVIDIA had a falling out. I do not know the state of it. Handbrake’s page seems to confirm this.
I am pretty sure it went thru h264videotoolbox
Maybe there is some software fallback in macOS.