The size isn't important, a T-rex had a huge brain but in relation to it's mass it's brain was tiny. The ratio of the brain size to the body mass (minus fat cells) is what's key - otherwise tall men would always be more intelligent than small men and being overweight would make people incredibly stupid. So height for height a man has heavier bones than a woman so he needs a larger head. Similarly, people with angular leg bones have less mass, height for height, than people with rounder bones, but they have longer legs, a narrower pelvis, a shorter spine, a longer neck and a smaller head.