The reward pool stays the same. If he downvotes $4,000 dollars worth, that isn't taken out of the reward pool, it just means $4,000 is divided equally among all the other posts. Think of upvotes more as shares of the reward pool than actual $. So, if the reward pool was $1,000,000 it will stay fixed. If every post and comment got downvoted to 0 except yours, you would get the full $1,000,000, no matter how small your upvote was.