You can already see who downvotes in the same way you can see upvotes.

Without downvotes, upvotes have the ability to just take money from the reward pool without anyone stopping them. Doesn't matter if it's shit content, without downvotes you can't do anything about it.

The problem is that we have a few very large voters voting for bullshit. If more of the small users would downvote it, it wouldn't matter. But since they won't, the users are taking $20k+/day in author rewards with no punishment or recourse