I struggle with this too! I worry about hurting people's feelings and sometimes this prevents me from organizing social gatherings. It seems easier to get together with no one rather than struggle with the guilt of leaving someone out. Did the other friends you were having coffee with ask why you didn't include that 5th person, or did it not even cross their minds? If they didn't mention it, that would have been a clue to me that I was overthinking the situation - which I frequently do. : ) Anyways, you are certainly not alone in this struggle. I'm right there with you!!