Just let people whine. You have the right to use your stake anyway you want, no one can tell you otherwise. A lot of people support other people being flagged, but when they themselves get flagged, they complain.

The flag is simply a downvote, it's no different than an upvote.

Keep doing what you're doing, the community needs more active flaggers anyway.