I wouldn't say INFJs never refuse hugs. I abhor physical contact by anyone I have not given my consent to.
Also, I have never asked a woman out, ever.
When I like a woman, I'm going to put myself out there, be available, courteous, and offer suggestions for things we might do. If she doesn't get the message, I'll say something like, "Hey, the thing on Friday is pretty yuppy, dress nice." If she STILL doesn't get the message/ask if it's a date/have the "define our friendship" talk/ask to fool around, I usually just pine over her in secret, and limit my contact to a more emotionally manageable, friendship level. That said, I don't have the worst luck when it comes to being asked out by women. Maybe it's a Minneapolis thing.
Not really sure why I approach it the way I do, I just don't like the thought of someone's company only being desirable, contingent on a guy's desire for a non platonic connection. I think there's a lot of pressure on women to be more than friends with guy friends, and I don't really like contributing to that pressure.
I have plenty of female friends, and have no problems with platonic friendships with women, though sometimes it might bug someone I'm seeing, and I can definitely understand why it might, having mentioned the pressures surrounding such situations, even though I've never and never would cheat on a romantic relationship. Regardless, I don't think I'm the only INFJ that can have meaningful relationships not based on romance.
Maybe you should ask him out. He may not have even considered you as a prospective romantic interest, if you all were friends first. If you are really good friends, and he's an INFJ, I don't think he'd be put off by you asking, in the least. If anything he'll just be curious why you asked, why you waited, what attracts you to him, etc... Either way, I'm sure it will be endearing, either as a friend or more.