Amelia - I have the same problem, and so does my husband. I don't know if it's because we're getting older, or if it's just information overload (we watch way-way-way more movies/series in the past few years than we ever have before), or some combination of the two...
It doesn't really bother me too much with Asian dramas, since they rarely have multiple seasons. But when it comes to English-language TV series (from America, the UK, etc.) with multiple seasons, stretched out over years, it really annoys us to death. If I can't find a detailed video recap of the previous season, we spend the first 2-3 episodes of the current season just trying to remember what happened before, who some of the characters are, etc., etc., and can't really focus and concentrate on what's actually going on with the episodes we're watching. :(
What we've started doing (if I can't find a good recap video), is to watch the entire last episode of the previous season to help trigger our memories. And in some cases, we'll even marathon the entire previous season before jumping straight into the current one.
It's truly annoying... especially since we're currently following 29 English-language TV series, and watching several Asian dramas simultaneously.