I suspect that's the problem with rear set application. Your lever on the gear shift is about 1/4 the length of the original, so the law of physics says you are putting 4 times the pressure on the bearing surface when changing gears. Also, the pressure is at right angles with that of the original lever, which has a balancing effect from the internal lever.
I*e when you push the original lever up, it causes the shaft to move upwards under load, but the internal part of the mechanism puts downward pressure on the pivot point. It isn't balanced because the internal lever is about 1/6th the length of the external one. AMC did get that right but you cant do much about the end to end rocking moment.
Have fun
Dereck.