Query: I've played a (small) amount with the multithreading
That is why schedulers go out of their way to keep threads on the same CPU (or CPUs), which means that even with SMP, multi-threading won't distribute work as evenly as forking.
on the bigger Sun boxes, and I don't remember having much difficulty utilizing the other processors on those boxes when I doing the multithreading.
Memory issues were a problem. We utilized Sun's multi-threaded malloc, which helped. But, for our application, most of the memory accesses were tied to a particular thread - so our memory bottlenecks weren't too bad.