if you have small increments in drill sizes it should follow the original hole.
try bluntening your drill bit first...it is a tried and true method of drilling materials which are prone to grabbing like brass etc even in a drill press.....
and if you think a blunt drill bit wont drill have a look at a masonry drill... just dont blunten it too much..it only needs the sharp edge removed so that it doesnt feel sharp.
be sure to hold the piece being drilled firmly in a vice .......