In the fifth chapter of his Sociology for the South, or The Failure of Free Society, Fitzhugh gives the pro-slavery argument I expected:
Now it has been the practice in all countries and in all ages, in some degree, to accommodate the amount and character of government control to the wants, intelligence, and moral capacities of the nations or individuals to be governed. A highly moral and intellectual people, like the free citizens of ancient Athens, are best governed by a democracy. For a less moral and intellectual one, a limited and constitutional monarchy will answer. For a people either very ignorant or very wicked, nothing short of military despotism will suffice. So among individuals, the most moral and well-informed members of society require no other government than law. They are capable of reading and understanding the law, and have sufficient self-control and virtuous disposition to obey it. Children cannot be governed by mere law; first, because they do not understand it, and secondly, because they are so much under the influence of impulse, passion and appetite, that they want sufficient self-control to be deterred or governed by the distant and doubtful penalties of the law. They must be constantly controlled by parents or guardians, whose will and orders shall stand in the place of law for them. Very wicked men must be put into penitentiaries; lunatics into asylums, and the most wild of them into straight jackets, just as the most wicked of the sane are manacled with irons; and idiots must have committees to govern and take care of them.
This harks back to Aristotle’s concept of natural slavery.