What I meant was that my self-esteem is mostly based on my values and beliefs. I deserve to be respected simply bc I am a human being, and based on the way I treat others. I basically won't let others make me feel inferior or take advantage of me bc I view it as unjust. That's the self-respect part which tries to protect me from the fickleness of others' opinions bc I know other parts of my 'self-esteem' are still affected by external sources. I also think some people have a greater ability to gain positive feedback and seek reassurance with good communication. We might think they are naturally self-confident but you'd have to be close to narcissistic to not feel insecure if you were criticised often enough.
I think I get what you mean about how you can feel insecure and self-conscious while still retaining a decent opinion of yourself. Maybe bc we are paranoid about being misunderstood? It's not that they will pick up on how flawed we are, but we won't have the social skills to defend ourselves and prove them wrong if they misinterpret our signals and deem us flawed, thereby exacerbating other negative social consequences for us. Perhaps it has something to do with Fe deeming the collective value system more influential than the personal, and how Ni's idiosyncrasies often are in conflict with their desire for belonging and acceptance. It does seem like we were taught to believe we have low self-esteem for caring what others think. I also think we are taught to believe that true self-esteem is the ability to respect and have confidence in our value and abilities, independent of what others think, which in many cases is a little unrealistic imo in a competitive, capitalist society, and even in a co-operative context...at least for many Fe users.
To elaborate, I don't have confidence in my abilities, but that's bc much of self-esteem seems to be dependent on external reinforcement of our abilities, and infjs are less likely to show them off to seek approval. So yes, it relates to what you said about not expressing yourself properly and living up to society's definition of a smart person in your example. You feel like you have self-worth and potential, but it will only be of any use if you can demonstrate your skills in a way that's recognisable and appreciated by others. You begin to think you will never be able to apply your insights into anything externally relevant, and self-doubt begins to cut into the realm of internal abilities, meanwhile having you resent that and wanting to defend your sense of shame against unequal principles simultaneously

At least that's how I experience the apparent contradiction.
When I am alone and no one is putting pressure on me to perform, although I might ruminate on some self-critical thoughts, I'm fine with myself most of the time (don't think I'm stupid or ugly etc) bc I'm not provoked to compare myself to others like I would in a social context. It's mostly a fear of being negatively judged but I'm introverted enough to know that external influence is not all my identity is based on. I may have 'weak psychological boundaries' but I also like being sensitive and empathetic. It's just about finding the right, most compatible context in which to implement the skills which might seem like weaknesses in more populous area- that is the hard part.