And make sure you have the hammer supported properly before whacking it!
I took a plain round gunsmithing block, that didn't have any holes already cut. I heated an old hammer,, to where it'd melt a bit of the delrin/poly stuff, into the shape of the hammer. Then, I drilled a hole larger than necessary for the plunger pin. Makes holding the hammer secure a lot easier.