I'm an INFP female engaged to an ISFJ male - so not quite the same, but perhaps a bit similar. You sound a lot like my ISFJ - committed 100%, dedicated, patient, thoughtful. I have to admit that I have had some of the same concerns with my ISFJ - feeling like we have lacked connection and that we don't always have things to talk about. My ISFJ has also said that I am not very good at bringing up topics to talk about! I suspect that fear of not being extremely emotionally connected and not being deeply emotionally intimate are probably very typical INFP fears. I know at least personally that I am a large part of those problems - I have a hard time opening up even though I want to connect deeply. He sounds similar.
Will he regret letting you go? Probably. But sometimes it feels easier to an INFP to just live in limbo than to choose closure. Closure involves all sorts of messiness... obligation, hurt, vulnerability. Reality not living up to hopes and expectations. Sometimes it's easier to keep the whole thing as a beautiful "could have been" rather than to experience its real challenges and pain. Also - was his last relationship very long? That could impact how much he is still feeling hurt/scared by it.
I of course believe that an INFP and ISFJ can make a happy and successful couple, but I think for the INFP commitment might end up being more of a leap of faith than for the ISFJ - for me it definitely was. Each time we have progressed in our relationship it has been a heart-judgment for me, more about trusting the wholesomeness and goodness of his character than pledging my ability to give 100% forever into the future. I do intend to do that, but it's not something I feel deeply secure in, not enough for that to be my basis for making such an important promise. I do however feel very secure in knowing when someone is right and good and full of love.
If he didn't have enough of that love/good/right feeling to outweigh the fear of commitment, then I think moving on was the right thing for you to do. INFPs are often slow emotional processors... maybe he is just not ready to open up to a new relationship yet.