Unable to remove my votes in the suggestions forum

Maybe it's me who can't understand how to do it but I can't remove my votes on a suggestion after voting. And if I gave "2 votes" on a suggestion and now want to change it to "1 vote", it says I don't have enough point to vote (even if I actually want to remove it)