I would suggest using care if/when 'using a punch to knock the pins out'. You can actually break the receiver where the detents sit by over zealous hammer/punch activities.
IF this is a continuing problem, push the pins out and look for the 'detent groove', It will be on the rear side of the pivot pin and the bottom side of the takedown pin. With pins in the 'out' position, put a drop on your favorite oil in the detent groove and then push the pin back into the lower. Set the lower in a position that allows the oil to gravitate around/into the pin recess for a couple minutes before going to the next pin. After the lube has soaked in, take some time and move the pins from detent stop in to detent stop out several times.
This usually helps/improves the movement. The pins are grooved and the detent divots made in separate machine operations which sometimes causes a burr at the edge of the detent hole. Continued battering will result in enlarging the detent pin hole OR as I mentioned, can actually crack/break the hole out.
I've seen a couple of AR 15's that were held together with socket head 1/4" bolts for this very reason.