The downvote should be removed just as you say here

I am not saying that. I was saying that for 6 months, and realized that if we did away with it that'd open the floodgates for other types of abuse. We can't get rid of it yet, as we don't have a way to stop the other things without it. So as I said the community needs to do what they can to remove support from people that abuse it.