We, that is my employer, has a web site for which I'm responsible. It is served by two web servers fronted by a load balancer. The web servers are Red Hat Enterprise running supposedly identical hardware except that one had mirrored drives. The really odd thing was that one of these webservers occasionally would get a rediculous load spike with a load average over 30. The highest we saw was 118. Yes 118. This particular web server occasionally ran an export routine that would process about 40 images and ssh them over to another server. The spikes seemed to corelate with when this routine ran. We niced it to a low priority. The load spikes still happened.

A couple of days ago my coworker Mike, took a look at the /proc/cpuinfo. The box that we didn't have a problem with said Pentium 4 3GHz and 512KB cache. Yep, as it should be. The box that was giving us grief said Celeron 2.4GHz 128KB cache. Ouch! Opened ticket with CoLo and complained. Next day the Celeron is gone and we have a 2.8GHz Pentium 4 with 512KB cache. Since then, no problems.

Moral of the story: the problem, as usual, was where it shouldn't have been...