Which enemy character had right to hurt weiyoung?

No one had the right to come after Wei. Her entire family was not only falsely accused but killed off and the people became slaves. 

Like Wei essentially said to Chang Ru, you can't lose what isn't yours. Jun didn't love Le like that and he even told her straight to her face but she didn't want to listen.