Very interesting! Seems like I don't understand probability as well as I think I do, since I bought the argument till I ran into this comment.
So, what are the effects of variance in different evaluation contexts, and do we have a meaningful way to measure bias if we take variance into account?
My initial reactions:
- It seems the higher the variance, the better examples you can trot out to say you are not biased against that particular group. since youll always find a member of that group who does amazing.
- If distributions of performance are multimodal, its even harder to conclude stuff because different institutions might cut off different modes when selecting the bar.
- Modeling the sources of variance may lead to insight into any actual bias.
So, what are the effects of variance in different evaluation contexts, and do we have a meaningful way to measure bias if we take variance into account?
My initial reactions:
- It seems the higher the variance, the better examples you can trot out to say you are not biased against that particular group. since youll always find a member of that group who does amazing.
- If distributions of performance are multimodal, its even harder to conclude stuff because different institutions might cut off different modes when selecting the bar.
- Modeling the sources of variance may lead to insight into any actual bias.