Does it turn? It just sits there, usually, I cant imagine a mechanical reason other than it is seized together where the bearing meets the tapered surface, and this should be free if its rotating and greasy inside. Might have a bent stub axle.
I prescribe a bigger hammer. A slide hammer would be ideal like you use when you pull an axle from a diff.