Trim the filter down to the part you only want to have affected:
If you want to do more than once part of the video note that you can add the same filter to the same clip or track as many times as you want and trim them down to the parts you only want to affect.