Making videos smaller - mp4 libx264 and libx265 options for additional compression

The Aim
I’m trying to achieve smaller videos without sacrificing quality, or having to reprocess in handbrake.

What I’ve Tried
I’ve tried to pass options to libx264 and libx265 to apply additional compression. Generally I would use the slower or slowest preset, and in x264 I would tend to use high profile at level 5.1.

I assume that these options can be passed to the encoder through the “other” tab, but it doesn’t seem to work as expected. I probably have the format wrong.

Details
With lib264, I’ve been using these options, pasted in just like this. These should compress things better than default, but I’m finding it makes files that are larger.

profile=high
preset=slow
tune=film

libx265 options I’ve tried “–preset slower”, “-p slower”, and “-p=slower” (without the speech marks). This doesn’t even export, it jumps straight to “done”.

Handbrake Comparison
When I export libx264 with no options, then compress with Handbrake slower, high profile, level 5.1, Q20, a video clip I’m working on now goes from 57MB to 46MB with no apparent loss of quality. Shotcut output to x265 is 32MB, Handbrake Q20 x265 takes it to 21MB.

So we can see there are gains to be made by passing in parameters, and waiting longer for compression.

The Question
Can anyone give me some guidance on how to export with additional compression?

This seems to work for x265, I can verify it compresses videos more than standard

–preset slower
–profile main

I don’t have a confirmed method for x264 yet.