Are you using the alignment tool? That might help a little. But there is only so much it can do if your source is varying over time
Your assumption that VFR is the problem might be right. It is best to work with CFR files in Shotcut.
Another source of the problem might just be inaccurate clocks in your recording devices. All source device clocks vary over time to one degree or another. Some more than others. To work around that, you could slice up the clips (once every minute, for example) and then let the alignment tool place the slices as close to the reference track as possible.