You have every right to be upset about this and to confront him, if you are worried about his reaction think of it this way, he is in the wrong, of course he'll be defensive and make out like you are over reacting. If you want to go along with that fine, if you want him to treat you like this then you can just say nothing and put up with it.
People treat you how you let them, if the shoe were on the other foot he wouldnt be happy about it either. If he is not prepared to cut contact and apologise for his behaviour then I'd be kicking his ass to the kerb.
If he's going to make you feel bad about his bad behaviour then that says it all for me, what a wanker and he clearly doesnt care about you.
+1 you said what I wanted to say but couldn't seem to word it