Two days ago at the range while shooting a .280 Rem loaded with 162 gr. Hornady bthp match bullets and (1) 51 grains Re19 and (2) 60 grains H1000 I noticed that at 200 yards both combos shot reasonably well about 1.25 inch, 3 shot groups. The 51 grain Re19 group was about 6 inches lower than the 60 grain H1000 group. Velocities for the H1000 load were about 200 fps faster than for the Re19 load and that velocity difference should have only made for a drop of about an inch at that range. Barrel vibes are the most likely reason. This .280 is a real strong bolt action -- don't try these loads in some Remington auto loader. In fact the H1000 load exceeds some load data in some manuals but is listed as max for Hodgdon data.
My guess is for handguns having short barrels recoil is the reason why these guns shoot lower with heavy bullet/powder combos than light loads. For me my, 1100 fps 240 gr. 44 mag loads print lower than my 1100 fps 125 gr. 357 loads. The 44 has much more recoil and is a SA Ruger Super Black Hawk, the .357 is a Taurus DA Tracker. Both barrel lengths are close to same at about 4 - 5 inches. The configuration of each gun is markedly different in regard as to they are held and pointed. My guess is that recoil is the only factor causing the difference in how they shoot.
Possibly rifles having short heavy fat barrels would react less to velocity and bullet weight changes.