I'm an ESFJ who has been dating an ISTJ for 1.5+ years now. I would not call it an easy pairing, and it has taken a lot of work, but once you put in the time and effort to construct that mutual understanding, it's awesome.
As ZDD and datMBTIguy already expressed, ESFJs need a lot of affirmation and they need it on a regular basis. I compare it to a plant that needs to be watered every now and then, or a car whose gas tank needs to be filled after a while. If we don't get it, it really takes a toll on our self-esteem and our faith in the relationship. ISTJs on the other hand seem to be permanently running on a full gas tank. The problem comes when they assume that the ESFJ is like them and doesn't need it. We ran into these issues earlier in our relationship (and still occasionally do). I expect more out of relationships than he does. He has to consider and do a lot of things he wouldn't normally do because I need them. I feel like this probably adds some stress to his life (and may be where your stress is coming from too). His relationship style is extremely low-maintenance. I could probably ignore him for a week and he'd be cool with it (if he isn't, he'd just find some way to rationalize why I can't talk to him and then be cool with it). I need a lot more than he does though. If he ignores me for a week, I'll begin to think he's lost interest in me and start looking for the exit door.
Communication is key on both sides. The other place where ISTJs and ESFJs clash is here, since the ESFJ's Fe means they expect other people to be able to pick up on their emotions without needing to tell them (since we're so good at this ourselves), but ISTJs are notoriously bad at picking up on people's emotions. If the ESFJ remains silent about the things that are bothering him/her, it quickly spirals into a cycle of angry-hurt ESFJ/clueless-frustrated-stressed ISTJ. I had to learn how to vocalize what I needed and what bothered me, and let me tell you how hard that is for an ESFJ.
We have both had to make changes. I have to tell him when something is bothering me. He has to remember to check up on me once in a while so I know I'm still in his thoughts. Once you figure out the right balance, the relationship becomes a lot easier.