The times I've been afraid on the street weren't due to catcalling - just regular old stalking or something. Nobody's ever told me I was hot and then like, grabbed me or something.
I think I do feel a lot more cautious if the streets are deserted, though.