Social ???? noooooooooo

I've noticed recently that I'm struggling with being Social. Most people I know get excited about a night out, whether just drinks with a couple of friends or a party, or other things like days out/group outings, or works do! But the idea of these things terrify me.
I'll put on a nervous smile and say great, thank you, that's a good idea, I sometimes manage a polite "not sure" or sorry I'm busy (its such a relief to be able to truthfully say this) the fight to not show the real terror and blind panic in my mind, is immense, desperately not wanting to be rude, but also just wanting to run away and hide. Changing the conversation, or looking for a way out, an escape route.
It has nothing to do with the person or people doing the inviting or trying to arrange the "thing" but its that social setting that is scary. In work, Zumba or even a salsa night, I have a reason to be there, things to discuss, from work stuff or the basic hello, how have you been? on the dance floor. You only have a few minutes when someone asks you to dance to worry about before the song has finished.
But the rest of the time I'm lost and terrified.
Its hard to put into words exactly what it is that scares me, it involves a few different things and they get all muddled together, spinning around and churning it all up.
Having nothing to say, having nothing in common with people, not fitting in, not understanding what they're talking about, or having no knowledge or experience of whatever they are discussing.
Being scared they'll actually learn stuff about me and realise what a "loser" I am, what an idiot I am. That they'll realise I really have nothing to offer or that is interesting about me, there's so little to like.
I feel I have no experiences worth sharing, my mind goes blank and I genuinely have nothing to say. 
If I do make it out I sit in silence, my mind running laps, grinning away and nodding, occasionally chipping in if I can pluck up the courage. Most of the time feeling silly and awkward.
I want to enjoy these situations and I can enjoy part's , but if I don't arrive or I do say no or someone you know does that, its probably not about you, or what your planning it may well be something in their head, for me its definitely fear and worry.