The problem with this perspective is that for each scale you measure people (e.g. how fast they can swim) a different set of traits contribute (e.g. shoulder width, torso to height ratio).
This means that you don't have a set of super humans that extraordinary at everything, but that for any arbitrary scale, the group with higher variance will tend to be over represented at the extremes.
One I've been pondering lately is suffrage. For most of history, and indeed most of US history, "men" could not vote either. A small percentage of rich white men could vote. The rest of us were in the same boat as women. And within one human lifetime from when all men got the right to vote, we gave women the right to vote. That's incredibly fast on social change timeframes. But the story is not taught that way in history class.
It's interesting how frequently this pattern exists where history is silent about class difference and loud about race/gender differences.
It was very weird having my last company's CEO talking to us about his focus on equity, when the mandatory company report showed his salary was 111x that of the average employee (at a tech company with no hourly workers).
At some point, you wanted to ask if he truly thought his salary was worth more than half of the development team's combined salary, and whether he thought that was equitable.
This is not simply due to what the 0.1% want, but nearly every person in the 99.9% likes “knowing” they are not at the bottom, and so choose to associate themselves with various classes within the 99.9%.
If you say "women do X" vs "some women do X", you're talking about women generally. I intentionally asked the question that way to highlight the irony.