I to have an American Predator converted to AI style mags. Since I have a Gunsite Scout I also have a pile of magazines. Old 3 and 5 round Ruger polymer mags work As do new model Ruger mags, Magpul mags and AI style steel mags.
Maybe call Ruger Customer Service?
It's probably Rugers kit DBA30452. I have one in the 18" 308 American that I got 2nd hand. I don't have this issue, so I can't really offer anything for advice. I have only a couple of the Ruger polymer AICS mags.
Reports are that newer Americans are machined differently that allows clearance for the AI mag, & older Americans are not compatible.
Otherwise, if just a little bit of clearance is needed, the mag housing can be shimmed enough to let the mag lock in.