I've always read that as Miles' Barrayaran prejudice speaking - in its own way just as bad as the sexism, only more invisible to Miles because he doesn't have a herm mother.

But of course it's never fully called out...