you need to be hitting down on the flange, perpendicular (or semi-perpendicular) to the housing. hitting straight on the end of the axle isn't going to do anything.

I know some people use a slide hammer to pull outwards on the shaft... but I generally have pretty good luck just working on it with a hammer and chisel/screwdriver around where the flange meets the tube.

yes, its really that simple, four bolts is all that holds it on.

-Rob