Have you tried using both Preview Scaling and Proxy? If not try them both at the same time and see if you still have the same problems. Click on the links below for info on each.
Also, just a tip, don’t bring the entire hour long recording to the timeline and then edit it down. Put it in your playlist and play it from there. Find the general sections that you want and mark the start and end points with I and O on your keyboard. Once you marked a general section of interest then bring that down to the timeline. When you bring those general sections down to the timeline then do the rest of your editing there.