There is no difference in the comparison because they're both based on immutable physical characteristics. There's a difference in the historical context, which you highlight.
This does not follow. There is a difference in historical context, which has a bearing on the real-world effects of height-vs-race discrimination, ergo there are differences between the two.
In exactly the same way: Abraham Lincoln and Pol Pot are both heads of state, but it is incorrect to say there are no differences between them.
You're describing a difference in the outcome, which is the action + the context. I'm not saying the outcomes are the same, I'm saying the action is the same, but the context is different. Subtle distinction, but it may not even have been what the person you originally commented to meant, so I am going to abandon further discussion on this :)
It's a comparison they are never totally the same. That's the point. If women complain they are treated like slaves, do you stop them to explain they aren't actually treated like slaves?
But for whatever reason we might say that racism has led to violent strife or wars, but has social tension between tall and short men led to war? Heavy conflict does not seem to organize along lines of tall vs short.
And on a related question, what is the essence of sexiness, and is attraction to the sexy wrong?