Do you know anything about it? Walton’s family wasn’t wealthy plantation owners. Whites who didn’t own slaves—which were the vast majority of them—were hurt by slavery.