How bout using a scope ring as the jig make a whole in it an then use a piece of stock? to make it stay put you can then center the whole in the ring, when its where you want put the bit through it
then you have no walking ,the picture is what made me think of the scope ring
Tommyt