Without a 5.7 mag to take dimensions on, I have no idea whether that setup would work. It might require some modifications to the bottom of the bolt or feed area to pick up cartridges correctly. Are 5.7 pistol mags single or double feed? A lot of variables here.
Re: pressure capabilities of the action, a calculation of bolt thrust from top-end .44M versus 5.7 tells me that the basic action is likely strong enough to handle the increased pressure in a small cartridge like 5.7. I can't be certain that the 77/22 components are similar dimensionally to the 77/44 ones, but given Ruger's manufacturing practices, I'd bet the critical areas are essentially the same.
I did come across some references to a similar action (Savage 40, I think; certainly a single rear lug type action in .22H) being chambered in .223, so I continue to speculate that the biggest hurdle would be making a repeating 77/5.7 feed from a magazine.