Anything is possible, but that seems unlikely
Of course everything is unlikely in this scenario. There are no likely explanations.
Yes, the database is remote from the webservers. Each webserver has multiple Apache children with independent database connections. The problem showed up on all webservers at the same time, across multiple children. It cleared up on all at the same time. And, as I said, it did not affect other tables, many of which have far higher levels of updates.
That makes me suspect that the problem must be internal to the database. Having the purportedly impossible happen once is unlikely. Having it happen 4 times in sync is less likely. Having it happen at a layer that doesn't have the knowledge to figure out when to work/fail to produce the symptom is again unbelievable.
This looks like what I'd expect if, say, you corrupted the internal memory in a validation routine, and then later that corrupted memory got reloaded from disk. (I'm not saying that that's what happened, just that that's something that could wind up looking like this.)
Cheers,
Ben
I have come to believe that idealism without discipline is a quick road to disaster, while discipline without idealism is pointless. -- Aaron Ward (my brother)
