Depending on the weight of the 50cal projectile, it may be safely possible.
A 660gr projectile weighs 1 1/2 ounces.
The heaviest shot data I have for 20ga is 1 1/4 ounces, for a slug 345gr.
If you can get the 50cal projectile down to 546gr, with the appropriate hull/wad column/powder/load data, it may be safely possible to shoot it in a rifled barrel or open cylinder barrel.
Now "if" all of this does transpire "safely", you aren't going to get but around 1150fps out of this experiment.
So whats the point?