you are pretty much describing what a locker acts like when it is driving one side only.
An open diff / air locker will help a bunch. I seriously doubt Ackerman would make much difference going strait down the road. My old rig would do this with a Detroit. It was really bad if one tire was lower than the other in back. New rig has arb in the back and drives like a dream with full hydro. I'd rather have a spool than a Detroit on the street for that reason.
It's not about the pressure, it's about the rolling radius. Get the thing on flat ground and measure it out. Read here: Excerpts from NoSpin/detroit locker® OWNER'S MANUAL
Mine would only dart around like that if the tire radius were mismatched from side to side.
Other things: As noted above your drag link and track bar are not parallel. You don't look like you have a ton of uptravel so I do not see why you have your drag link mounted underneath the steering arm. Flip it (get real hardware for it as well, not Nyloc nuts), and see if that helps with on-center feel. This will also help when the steering droops out so it doesn't try to over-flex the rod ends. Also get some safety washers or something for your tie-rod ends.
So what happened was the tires on the rear got chewed to hell in short order from a welded rear. The front tires are like new but one of them got a sidewall cut and got replaced at some point. So I rotated those straight back. Theyre very similar in depth, I'm not even sure which is which at this point. I'd have to measure. Didnt think it would throw that grizzly into a fit like this.
They are parallel at ride height, the picture is deceiving but even based on angle finder they are the same.
I honeslty cant recall why I mounted my panhard in the middle of the 3 holes on both panhard mounts. I had thought about dropping it and maybe raising it on the axle side and then flipping drag link to top to get matching angles.
The washers are probably a good plan, but to me the bolt heads looked bigger than the heim bodies... Not arguing there though. I have mechanical lock nuts on most of my hardware. Whats wrong with nylocs other than one time use?