Maybe there should be a limit to how much a certain person can damage another one's reputation. By doing that, flagging someone down would really be a community effort.

Then again, there are communities of spammers (!) with significant reputations who could counter a flagging of one of their own...

It's really tricky.