And I will agree that there are sex differences over the population of each sex -- abilities (and inclination) aren't distributed equally, just as the distribution of height in males and females differ.
However, people do forget that not only is the upper end -- "geniuses" or top scorers -- has more males than females, but the lower end has more males than females, too. If you go into any special education class, you'll find more boys than girls (and not just because of behavioral problems). More males are autistic, more males are dyslexic, more males have all sorts of mental development problems. So what is it about males that makes them better =and= worse than females in math (as a group)?
Let's pretend the population means for some measure of math achievement are the same (I don't think they are, but that doesn't matter here). Look at two normal distributions with the same means, but different standard deviations -- the male population having the higher std.dev. If you look at probabilities - like how many of each sex score higher than =blah= (something above the mean) - you'll see that there are more males than females... the farther you go out in the tail, the more disparate the probabilities. Of course, if you explored the lower tail, you'd see the same thing. Even when the population means are different, if you go far out enough in the tails, you'll see the same behavior. In general, you'll find the difference in the population means is smaller than the standard deviations in each population -- and that's true whether you look at it sliced by sex or race. Of course, Summers being at Harvard means he's only seen the upper tail, not the entire distribution. And yeah, since the tail for males is fatter, you're going to see a higher mean for males and more males in general.
In any case, what's that got to do with the price of tea in China? There's a second factor here, and it's something called =inclination=. Consider me - I did extremely well in physics and math in undergrad - in fact, probably better than all the other people graduating from NCSU at the same time as me. Now, when it comes to fields, females are way underrepresented in physics, and somewhat underrepresented in math. But my inclination was toward math. Should I have gone into physics to make their numbers better? Then, in grad school, I got my masters and dropped out before I got a PhD. Did I have a duty to other women to stick it through and get the PhD whether I hated it or not? I knew there was no way I was going into an academic career -- the pay and hours suck, not to mention the whole irrelevancy of most research to all but a few people. And I wanted to have kids, and a husband to stay home to take care of them. I wasn't going to get that in the academic biz. I have a duty to myself, and my family -- I recognize no duty to femalekind. I don't see why I should do something just because I'm female.
So I jumped ship to the far more lucrative corporate world. I imagine that lots of other people from "underrepresented" groups also try their hands at academia and finding it wanting in a practical sense and go do something else. We have no duty to fill niches that we're not interested in, no more than do men have the duty to fill in where they're underrepresented, like as preschool teachers. This is not a failure of society or of education (in my case, at least). But, of course, those in academia think they're in the highest echelon of whatever, so of course the best of everybody must be there, and there must be something horribly wrong if the smartest people decide to do something else. Whatever. Not my problem.