What I mean is referring to genitalia as a pussy is not inherently sexist, only vulgar (as opposed to calling a guy a pussy because that implies he's woman like in a negative manner). But in this context, taken as part of the situation and also a longer term pattern of behaviour, I believe his use of the word in this case, regardless of the fact that it was referring to genitalia, was in fact misogynistic.
EDIT: The corollary I'm making here is that calling a gentleman of East Indian or Pakistani descent a "paki" is inherently racist. The word "paki" itself is inherently racist. But the statement "Women have pussies" is not sexist, it's just a crude and vulgar statement of fact. But the situation in which Ford used it in today, even though he was technically using it in the same sense as a descriptor for a vagina, was in fact sexist because of the specific context of the situation.