The difference is entirely a quantitative subjective judgement: How likely is this person to harm me?

There's no hard, ethically relevant distinction between the situations. Law can and does distinguish between them, but that's because law is in the business of drawing arbitrary lines in the sand (as it must be).