I was in a relationship with an infj for 3 years.
He was insanely possessive, neurotic, and jealous.
He probably doesn’t reflect the type, but he did teach me that even imaginary breaches of trust (I am and was very faithful until he raped me) have an impact.
If the trust isn’t there, he probably isn’t for you.
It is a matter of matching values.
If you need someone highly monogamous who never makes you feel doubt, that is ok.
It appears this man cannot, that is all. It is a case of mismatched values.
Talk it out and see if you can get closer, or end it.
Or cheat on him and get even.
That would be hilarious.