Toyotas are fun and don't need a whole lot to be truly offroad-capable. The only downsides are if you enjoy having a powerful motor, or any interior space whatsoever. The 20R/22R/22RE are good for reliability and low-speed torque work, but kind of lose their luster when high RPM's or cubic oomph are called for (mud, sand, highway). There's also the matter of stock-odds being that you'd have to either deal with a carbed motor or IFS, unless you find that single year ('85) when they put the solid axle with the EFI.
XJ's downside is the relatively weak axles for weight and power... power, that is, if it's a 4.0L 6-cyl model. The 4.0L is arguably one of the best motors around in terms of its mix of power, weight, dependability, and durability. If it's a 2.5L 4-cyl, only buy it if it's REALLY cheap because it will be an absolute dog to get up to speed, and the manual tranny with those models is unimpressive. XJ unibodies can actually take a good bit of abuse, and your D30 and D35 (if the rear isn't a 8.25") will go before the unibody does. The D30 a '93 would have (HP, non-CAD, 297x joints) is amongst the strongest representatives of the type. They also need a bit more lifting to clear bigger tires than the Toyota or F150 would.
The '78 is (IMO) one of the best eras for the F150... HP D44 front, 9" rear, and typically pretty stout drivetrain overall. The only obvious downside here is its sheer size. Being the largest in your prospective wheeling group, you may either find yourself kept out of tight two-tracks that the others are taking. The carbed 351M or 400M aren't highly thought of since they're thirsty, heavy, and underpowered for a V8, but they're workable and easily replaced by a 460.