Yes you perfectly sum of the experience of people with actual moral principles. I keep seeing this problem framed in terms of money you "lose" by not being selfish. But that assumes you actually have, could have, or should have that money. As you point out, it doesn't work that way for people who see the truth.

There is a lot of money in prostitution, so I hear, and few barriers to entry. I have some idea of the caliber of prostitutes in my area, so I know its not necessary to be young or beautiful. But it doesn't fit with the truth of who I am and what I want for my family, so I never think about the prostitution income I am losing. I was never going to have it anyway.