It sounds like it could be the spindle bearing.

There is a small needle roller spindle bearing that installs from the back side of the spindle and supports the inner part of the outer axle stub.

It is a pain to get to, a pain to get out but that is probably what you get to do.

Good Luck,
Michael