By any definition this is a derogatory term (rabble, riff-raff, shabby, motley, dishevelled, etc).

I think it's a derogatory term.

One way to render it impotent is for the group in question to use the supposedly derogatory term for themselves.

They seem to think it is a derogatory term which undermines the dignity of their ancient and prestigious post.

It became a derogatory term during his peak years in football.