I understand that you (and by this I'm not addressing any particular person, but the whole of you reading this) don't always want to play against a specific opponent. That is fine, no problem there. But please people, start rejecting if you don't want to play. 
Only 1/5 of you (I'd like to say I'm exaggerating, but I'm probably understating this) actually hit the reject/refuse button. The remainder sit there like you are afraid of hurting my feelings or something. 
Some of you are actually away from your computer, I get that, but there's no way 80% of you are AFK. Please speed up this process, otherwise you're just wasting both of our times. I promise you I won't take offense.