Unsure of the context, but usually, the issue is people applying subjective understanding of 'privilege.' Most of popular skirmishes around this word stem from misunderstanding White privilege (by Peggy McIntosh) which is watered down & applied in other areas.
In your example, it might be useful to ask the people you're talking with: what do you mean when you use this term, privilege, and how do you see it different to a right in this example? Goodnight!