If you're using a cheap bit from a multi-bit 1/4" drive set, it's very likely to fail when attempting this operation. An impact bit may work better, best would be a high-quality Torx-specific set or Torx driver of the appropriate size.
Also, sights are a typical place for the application of threadlocker. A little heat will burn it off and lower the force required to remove the screw. If you think this Torx is bad, consider the 0.05" hex grub screws used to retain some sight components. With threadlocker, those will round off or strip out every time.