Social class in the 1850s, when slaves were a primary source of labor, was very segmented. Class mobility was very limited and social status was determined by race, gender and wealth. Regardless of the large differences in class, there was limited conflict because they all depended on one another to function.
As a member of the planter elite, I hold the most power in the South. I own 40-50 slaves and 800 or more acres of land.
I am a small planter. I own zero to one slaves. I wish I was a part of the planter elite but at least I'm not a woman.
I am a woman. I am subordinate to the men in my life, but at least I am not a slave.
I am a slave. I live a life of social inequality because of my race. I am treated as property and am owned by another human. But at least I am not a hills person.
I am a hills person. In some material aspects I am worse off than a slave.