All a security guard can do is ask you to leave private premises.
It's worth carrying a print-out of Andrew Nemeth's cheat-sheet on NSW photographers' rights. That way, ignorant fools can be educated.
You'd have to make a judgment call based on the situation.
But really, photographers, who are pursuing an inoffensive, perfectly legal and socially acceptable recreational activity, should not be hounded by ignoramii and made to feel like predators and criminals.
The more we bend over to the ignorant masses and cop it, the less freedoms we will have.