This is exactly what I've been meaning to post about.
For the record I totally relate to you. I've been super hard on myself pretty much always, or so I've been told. I have pretty high expectations (of myself and not that much of others, mind you) and I'm totally a perfectionist as well.
I think it might have a lot to do with the Judging preference combined with Si. Also, something that really stuck onto my mind that I read on this forum a little while ago was that ISFJ's seem surprisingly uncomfortable about their feelings around other people. Definitely true on my part. The mindset may not always be there – but often I feel like showing my emotions around others makes me defenseless or vulnerable.
So seriousness makes me feel a lot more self-assured and prepared for whatever. I do wish I could ease up sometimes, though. I have a lot of laid back, care free friends who are constantly telling me to not take things so seriously, and kind of hint towards wanting me to loosen up a bit. I do feel a bit misunderstood, though, to be honest. Being serious is almost like a natural way of behaving for me, so having my guard up is just me going with my instinct.
I relate so well with what you're saying. I've picked up on "laid-backness" over the years, enough where it can
appear that I am a laid-back person most of the time--and nothing wrong with that--but I am hands down a person who takes life more seriously than "I should." Like you said, whenever I hear that, I feel misunderstood because just as it is another person's natural inclination to be nonchalant and perpetually relaxed, it is
my natural inclination to mull over things, consider situations thoughtfully, and take on life in a serious manner. Honestly I don't think there's anything wrong with it--balance is key, but I'm definitely coming to peace with who I am naturally, at the core.
One question though: can you elaborate on what you said about ISFJs being uncomfortable about their feelings around other people? I think I understand, but I just want to make sure.
Because when I'm serious and they reject me it's only a defensive facade they're rejecting - not the real me.
Seriousness is my safe place in an unfamiliar environment.
I decide when and where I let my guard down although it happens more frequently than it used to. Perhaps this has to do with our introversion, because what use is wasting our limited effort extroverting with people we don't like or care about or in an environment where we are very uncomfortable?
That's it! From what I'm gathering, our *general* seriousness--around people we're not comfortable with--is equivalent to us having a necessary wall up, to protect us from
whatever, which brings me to...
interesting...because the ISFJ i know best is one of the goofiest, silliest, people i know. he's serious when it comes to his responsibilities, but his general outlook is easygoing, neither pensive nor serious.

i'm wondering if maybe seriousness or the lack thereof is more related to enneagram type than to MBTI type?
...when we are around people we trust/are comfortable with, we--or at least I--act extremely different than our usual serious exteriors, often in silly and humorous ways. Am I right?
@Emerald_sea: Not to say that the ISFJ that you know can't naturally be easygoing, but I think his/her comfort level with you has a lot to do with being goofy and silly. I'm the same way with people I'm close to, but around people who don't know me: extremely reserved, solemn, and purposely hard-to-read.