IMO, it's because he sees her and the "past her" too differently. He likes her for who she is and his past recollection of her doesn't align with the way she is now, so he lets it go.
I didn't look for a BL plot but it seems this one was made and got censored, judging by the end credits. You guys are not crazy for shipping them two guys since it's the creators' intention. It's sad to see it got censored, a lot of the series was as it seems.