Are those Tamer Grips?? They should have a Smurff Blue Sorbothane insert down the back on the inside. If they are standard Hogue Grips, you may want to upgrade.
Reloading is a big help. You can shoot enough to get good.
I have a 12" X 12" Steel Plate out in the pasture hung off a HD Saw Horse to practice on with my 480.
I wear the Howard LeHigh Impact Sports Electronic Hearing Protection while hunting. They improve my hearing a great deal, as well as protection from the Loud Muzzle Blast of the Big Handgun.
The 454 SRH should make you a great deer hunting handgun. You may want to use it more than for one day. I have not had a rifle in the woods in 3 years.
Bob