“Let him who expects one class of society to prosper in the highest degree, while the other is in distress, try whether one side of the face can smile while the other is pinched.”