People make vertical videos with Shotcut all the time, so it is possible. However, changing the Video Mode on an existing project can have many unpredictable effects on filters that were set up for the old resolution.
Try creating the custom Video Mode at 400x700 and starting a brand new project in that mode. Then bring in the vertical video. It should “just work” without any filters needed, unless the vertical video and the 400x700 resolution happen to not be the same aspect ratio. Then the vertical video would have to be cropped to fit a 4:7 aspect ratio which would then eliminate the black bars. This might even be the problem your current project is having. 4:7 is a very unusual resolution and aspect ratio, so I would expect some cropping to be necessary to remove black bars. That, or a pillar blur background in place of the black bars.