@mjavedva, have you seen the research I posted at Producing minimal, Youtube compatible, file sizes - #2 by kagsundaram ?
The three things I found that reduced file sizes were:
- Lowering the frame rate
- Lowering the Quality
- Raising the GOP size
The first two may be non-starters for you, but have you experimented with a larger GOP? By doubling my GOP from 150 to 300, I had a drastic file size reduction with no noticeable decrease in quality.