Yeah it seems a necessary flaw. They system would work perfectly if people didn't naturally take offense to the downvote and refuse to use it because of that potential backlash.