krwheel is right.
That is one key difference, I am terrible at remembering the details of something years after the fact. My ISFJ friend was telling me the plot-line from the original Star Craft the other day, because he actually remembered it!
There are actually a lot more INFJ's out there; we're probably not that rare, just we often test as something else. See, with introverted sensing, which is what the ISFJ's have, you have to build your inner world on facts. With the INFJ, you don't have to use facts to build your inner world, you can build WHATEVER you want. This results in INFJ's being highly diverse individuals who usually do not see eye-to-eye on things. ISFJ's are very likely to see eye-to-eye on things, unless their inner world of sensory details leads them to different conclusions. Even then, it's really just a matter of reconciling facts with each other, and at the very least they could relate to each other's point of view after doing that.
An INFJ's inner world is unique, and it's doubtful anyone else could relate to it. Often, we may not even know why we think what we think because we came to a conclusion about it already, and the details that led us to that conclusion are irrelevant. It is the opposite for the ISFJ, the details that caused them to reach the conclusion are very important, and they'll remember them.
I could tell tell people the entire plot line of some games i haven't played in years.
Before knew i was an ISFJ i was perplexed when people couldn't remember conversations i had months ago with them. This next thing scares some people my mind burns in personal information of my friends if i'm told something from one of them. I had a friend i knew for 8 day and he was curious about this so i wrote everything about him i had learned in the last 8 days. I wanted to take it back so bad after i gave it to him and starting apologizing for it. he said this is pretty cool and asked me somethings about himself and i knew the answer of those on the spot.