That is surprising. Usually, if audio sync gets off, it stays off or keeps getting worse.
One possibility is that your source material might be variable frame rate, or your container might be difficult to seek. As a test, you could try to convert both source clips to an edit friendly format.