Short answer: there is nothing wrong whatsoever with being single...
I am 36 and still experience feelings of reluctance when it comes to becoming involved in a relationship, with or without chemistry. You need only weigh your need for personal space and independence against your desire to be with this person. In this instance the choice seems simple. There is not enough chemistry to justify getting involved in something that could potentially become serious and difficult to break off when (and if) the time comes, and in this case that it seems likely that it would. Of course this is contingent on the young woman's desires as well, but...
Then again, there is the option of casual dating, though I have never been good at this, and I suspect our INFJ counterparts are the same. Nevertheless, I have done it and this could be a solution. Every encounter need not end in a deep, meaningful, time-consuming relationship. You could explore the situation you mentioned (especially considering that it is your first date), see if it leads anywhere, and not allow it to get any more serious than you are prepared to handle. This could satisfy your desire for human companionship, while simultaneously allowing you to maintain your overriding desire for solitude. If it gets more complicated than you want, simply opt out. Of course, if it's wrong and you have absolutely no doubts that it's wrong, then I probably wouldn't bother.
I am an enormously independent person, though in my youth I was lonely and hungry to be in a relationship. I have been in a few relationships over the years, mostly short-term, and the overwhelming majority (at least in my youth) have been something I rushed into out of loneliness, curiosity, desperation, etc. These should be simple to terminate, though I inevitably feel guilty when it comes time to end these relationships, even though I knew they would likely not work out from the beginning. Of course I do not speak for all of us, but much of my reluctance to date casually (or even to invest in a relationship) stems from this. My natural proclivity for conflict avoidance, innate empathy, and inability to share deep emotions does not make this any easier.
Of course, not all relationships are of this variety. Sometimes, even with a relationship with all the necessary chemical components in place (of which I am not entirely unfamiliar), there is still that underlying desire for personal space and independence...even if it is great. This does not mean that you wish to run around and date other people, or that you do not love that person, only that you prize, and likely require, solitude. This is not a flaw, not something you need to change, unless you want to, but can definitely lead to an intense feeling of dissonance.
It is quite natural to wish to be in a relationship, or simply to desire human contact and all that comes with it. It is equally natural to desire your alone time, and, let‘s face it, our temperament requires no small amount of time to recharge, contemplate, and compose our “masterpieces“ (whatever these may be). I cannot speak for all INFJ’s (or even introverts), but I need ample amounts of “me time,” and this requirement increases as I grow older. By “me time,” I do not mean I need a couple of hours each day. I need days on end of uninterrupted time alone, without the impending feeling that I must squeeze it into some narrow framework. It is in my nature, and, to be honest, I have no wish to change it. Constant contact, even with a significant other whom I love, exhausts me. It has nothing to do with that other person. It is not personal, it is just the way I am.
You can change your nature (or at least its outward manifestation) to a degree, but you should only feel compelled to do so if the unfulfilled desire for companionship is causing you more distress than being alone is bringing you happiness, and certainly not because society tells you that the only way to happiness is though human contact, or because someone else does not like or understand who and what you are. Otherwise, exercise patience. You can find a balance, but the end result should make you as close to happy as is possible, and should not feel like a compromise.
This is undoubtedly more than you wanted to know, but this is my take on your question.
“Well you know that I love to live with you,
but you make me forget so very much.
I forget to pray for the angels
and then the angels forget to pray for us.”
-Leonard Cohen
