Before I'd alter the firearm,,, I'd work on things that are changeable & fixable. Different reloads, different bullets, & after you have a few different good groups,,, let a few other folks try them. Especially folks who know how to shoot a SA.
I have a tight shooting Vaquero in 45 Colt.
I tried many different loads, and many shot very well. HOWEVER, I tended to get "low & left" a lot. So, I got a few others to shoot it. One is a respected handgunner I call my friend. He claims I "sandbagged" him, as he shot a tiny one hole group with it. (I didn't tell him how tight it shot,,, just that it didn't shoot good for me.) BUT,,,,,, it was low & left. THEN I knew it was the gun. So, we did some minor alterations & it is more centered & level.
Just trying one load, or at one distance, or only one shooter will not give you enough info to worry about a gun problem. As a wise old gun person told me once,,, long ago; "Many gun problems are the fault of that loose nut behind the trigger."
To that end,,, I've worked hard at removing all the mechanical or ammo related issues before I shoot so I know that when I miss, I can only blame myself. Some guns do great w/o any issues, and others I've spent a lot of time working on. Eliminate the stuff you can easily change before cutting metal.