The thing is, a lot of us like to think of ourselves as being "nice" people... and some part of our social conditioning tells us that "nice" people don't just say no without a "good reason." So we have a tendency to fumble around for excuses and rationalizations, as if simply not wanting to is not a legitimate reason to not do something.