Why, i always get confront photography in public not other?

I was at the Olympia park taking pic and i was confront by security. he ask for my pass which i show it to him and he ok, but what i see two people in organe jackets in not a security or polices come from behind me. i feel it a aggresssive and interminitation. and there a few people taking pic and nobody ask them a word? this the man i was talking too.