People taking such myths seriously is the crux of the problem. After all, if you were born low because of your alleged past sins, why should you not suffer in this current life?
Well, atheists don't believe in anything before or after this life. It's usually such people who have no fear of committing crimes when they are confident they won't get caught.
> why should you not suffer in this current life?
they will suffer. But not because of you or me. But because of reactions to their actions long ago.
Does it mean you can ill-treat them without consequence ? No. If you do so, you will be born in that caste in a next-life and suffer the same oppression they're facing now.
The Hindu caste system talks about spiritual evolution of the soul as it goes from body to body in each birth.
You might be a lower-caste in this birth, but you might get upper-caste birth in a next life. And vice-versa. That's how you escape it.