Good advice by Julius Motal at The Phoblographer:

Belligerence towards an unwilling subject in street photography is at the very least unwarranted and deeply disrespectful. It signifies a disconnect, a lack of empathy, which ultimately affects the image and the photographer. Frustration is a very real and natural thing to feel, but when a photograph goes untaken, it’s gone. Nothing can really be done about it, and when someone signals that they want no part of it, it’s best to let it go.

Good photography is predicated on respect for your craft, for your peers, and most importantly for your subject. How you carry yourself and your camera on the street ultimately affects each image you try to make. If you’re aggressive with your camera, people will be inclined to defend themselves. Aggressiveness doesn’t necessarily mean exaggerated actions. It can be subtle. How you react to someone who objects will determine how the rest of the interaction goes.

I never feel anger towards my subjects, but I’m no stranger to feeling frustrated with myself. In my case, it’s often due to me hesitating a split second too long before pressing the shutter release. By the time I muster up the courage to take the picture, the moment is usually gone. The feeling of knowing you could have taken a great image but you were too indecisive to go for it is very real and can frustrate the hell out of many budding photographers. I suppose I’ll get better at judging these situations and going for more shots as I spend more and more time shooting in the streets but for now, when in doubt, I still prefer to err on the side of caution.