You aren't being silly. At least I don't think you are. If anything, that girl should have been the one to get up and take your roommate with her out of the room if she really thought they needed privacy. It's your room, too, it's so rude to ask you to leave. If your roommate herself had done the asking, I would understand, sort of. But if anything, yeah, your roommate should have at least notified you when it was safe to re-enter. Roommates! They so crazy, sometimes.

One of my old roommates used to bring her bf into our room ALL THE TIME. Whether or not I was there. And he'd spend the night. SO AWKWARD. Hopefully your roomie doesn't do shit like that. :(