I think this phrase sums it all up: "The story of my life: stuck in adolescent stage fright when it comes to women". There's James' dilemma, some bad experience marked him forever. Sometimes, in the first loves of youth, the whole heart is left and there are no more leftovers for anyone else.