Let me explain a bit...
The more introverted INFJ would simply have a less developed Fe and choose to fall more so into their Ni, but they still very much so care about people and their relationships -- the desire and need is still there because they are INFJ. I doubt an INFJ could just completely not care.
Just as the more extraverted and outgoing INFJ can APPEAR to not ever be introverted and need that alone time... but because they are INFJ, the base of their personality is the same as any INFJ so they still need that.
I do very much so agree that a major difference comes from Fe, but just not necessarily in our needs and desires... its shown more how others see us.
That being said, I think as INFJs we are capable of easily jumping between extraversion and introversion.... its just whether we choose to or not. I know for me, I just need a push by others, job, or the environment in general to be more extraverted.
It's funny because at the moment I'm probably more on the introverted INFJ side than extraverted, but I know I could never have a job with no contact with people, which is seemingly a contradiction to some people. I would seriously get depressed sitting at a desk analyzing data or whatever all day. I need to be with people and helping people to be happy.