Some rear sights are fitted so tightly they will not budge without significant brute force.
This is just an anecdote but I had to remove the rear sight on my S&W Shield to install the slide portion of the Apex trigger kit. I had to custom cut and profile a 1/4" chisel to fit the ledge of the sight (maximum surface contact to minimize marring) but even before that it has a thread locked set screw that must be removed, then I froze the slide assembly for several hours before I made the effort to remove it.
I have little doubt that Ruger rear sight is similar. They use sight moving jigs and they can force the sight in the dovetail so tight it's freaking amazing what it takes to move them.