First of all, in all the self-defense and security courses I have been given (courtesy of the Federal government), "situational awareness" has always been stressed as your #1 defensive tactic. I would NEVER turn my back on someone in such a situation!
Secondly, I am 73 yrs. old, with a heart condition, diabetes, and "brittle bones" (osteopenia); I would be "in fear for my life" where a younger, more active person would not. I agree that no one can answer for someone else but, if he appeared to be pressing the attack, I would drop him.
All of the above assumes my normal evasive, conciliatory remedies did not suffice. I am NOT Wyatt Earp!