There should be some slop/play in the center bearing mounting holes when the bolts are loose. Try and move it so the driveshaft looks almost inline from directly below.

Also check the pinion flange for play, bad pinion bearings could cause this too.

Andrew