This is not supported at this time, and it is not as simple as writer a shader file. Code is required in MLT and possibly Movit to support it. Also, Shotcut uses a pinned version of Movit that is rather old because newer versions require a higher GL context version that is incompatible with the Qt technologies Shotcut is using. I might switch back to using Movit git master and giving it a dedicated, non-shared GL context with the version it needs at the sacrifice of some optimization (render GL texture to image in CPU RAM instead of integrating Movit’s GL texture into display scene graph).
I see no easy way of making Movit extensible without having to make your own MLT plugin. Easier to use WebGL for now. Support for WebVfx transitions is coming.