He's particularly peeved because he can SEE how he's been manipulated, even when it's for his own functionality. Like: he really doesn't LIKE to think about how Illyan dug up his mostly buried need for approval and then fixed him on Gregor for it, but he's *aware.*