Well here's a fun one.

We have an application that is humming along on high volume against Oracle 8i. Normal (fairly low) levels of errors (mostly due to robots submitting bad data).

And then 2 web pages start having problems. Not every time you try to save data on them, just 80% of the time or so. If you're lucky enough to get a successful save, the write really does make it to the database. And it doesn't depend on the data, if you hit "Refresh" enough it would eventually save. Luckily they are low-volume pages, not a lot of people see them. Unluckily we tend to care about the people who see them, they are walking distance from the programmers and we try to be on good terms with them.

No code changed on disk. No ddl happened on the database. The application logs show tons of ORA-06502: PL/SQL: numeric or value error messages. (You know, what you'd expect if you tried to save text to an integer...) No errors on Oracle's alert log, from the operating system, or from Veritas.

3 hours later the problem stopped on its own.

So what to do now? Try to escalate it when we have no diagnostics indicating what happened? Or sit tight because we're planning to upgrade to 10G anyways shortly?

Luckily the decision is out of my hands, but I've never seen anything like it. (Nor do I wish to see anything like it again...)

Cheers,
Ben

Of course everything is unlikely in this scenario. There are no likely explanations.

Yes, the database is remote from the webservers. Each webserver has multiple Apache children with independent database connections. The problem showed up on all webservers at the same time, across multiple children. It cleared up on all at the same time. And, as I said, it did not affect other tables, many of which have far higher levels of updates.

That makes me suspect that the problem must be internal to the database. Having the purportedly impossible happen once is unlikely. Having it happen 4 times in sync is less likely. Having it happen at a layer that doesn't have the knowledge to figure out when to work/fail to produce the symptom is again unbelievable.

This looks like what I'd expect if, say, you corrupted the internal memory in a validation routine, and then later that corrupted memory got reloaded from disk. (I'm not saying that that's what happened, just that that's something that could wind up looking like this.)

Cheers,
Ben