First, I would have told him that I stopped listening when he called me a liar. Second, while a negative cannot be proven, a positive can. I would have told him to prove what he was saying or get lost. OR, if you were physically uncomfortable with the situation ( numbers don't necessarily mean anything) you could have simply walked away. If he would not allow that, respond as best you can, and make sure you have a good description if not a name and charge him with assault ( if your response wasn't adequate, or even if it was.) Unfortunately there are a lot more folks with sociopathic tendencies running around due to a certain kind of upbringing. No quarter can be given. Also, as he is not a community member (or even if he was) he is of no importance and should be treated that way.
Btw- there is no age limit to becoming prolific in many hand-to-hand styles. Trust me on this.
Lou Ferrigno in his 70's