Well, if you want the enneagram answer it's basically a mix of feeling like you might be incapable of dealing with the external world and that the external world might tax you more than you think you can be taxed - like it will make a demand on you that you can't meet. It's the mistaken notion that we're separate from that world.
Having said that, I wouldn't say I have much social anxiety at this point in my life and I'm wondering if that's more what you're talking about. I'm withdrawn and I spend a lot of time alone, but I also do things in my life. I'm a pretty active person but a lot of times I like to do that activity on a solitary basis or with a close friend/partner/family member. I spend a lot of time at home in my "castle" but I spend that time doing activities I enjoy and I don't feel like I'm missing out on the outside world.
In my opinion, there's nothing wrong with being withdrawn and wanting a lot of alone time so long as you're comfortable with it and maintaining some relationships. I used to think I should be more social or have more friends but I'm happy with the few I have and I don't have trouble being social when I do go out (which isn't very often). If you're feeling like you're not participating in your life though, that's another issue. It sounds like you feel there's things you should be doing/be able to do that you can't. While anxiety will never fully go away (and it shouldn't, it definitely has it's purpose), there are a lot of things you can do to help deal with it and overcome it when it might be holding you back.