inexcusable

Post #315,623 by Steve Lowe 10/14/09 12:25:23 PM Reply	inexcusable Basic risk analysis would point this potential weakness out. We won't TOUCH a datastore or database server without a well-defined maintenance window with risks and contingencies identified, and the first step after taking the database offline is a backup. The risk of making the backup is infinitesimal compared to what they're facing now. Also, any fool who'd design a service like that without a) at least one georedundant data center, and b) a separate geo-redundant backup system should be hung from the nearest yardarm. Likewise, anyone who'd create this task without a known-good backup as part of a fleshed out contingency plan should join him. I've seen a fair number of articles (mostly written by idiots) claiming this is yet another reason that cloud computing is teh bad. Personally, I don't think you can apply that label here because this service is apparently only slightly better equipped than the high school webgenius with a stack of servers in a back bedroom cooled with a box fan and served from wifi stolen from a neighbor. As a company that sells services based on cloud and grid computing, that kind of reporting is what pisses me off. Yes, we're georedundant on backups and content delivery, and will be on application serving within a year. And no we don't have on the order of 11MM users, but we apparently take access and security of our client data a hell of a lot more seriously than Danger.
Post #315,624 by drook 10/14/09 12:29:24 PM Reply	Little harsh there, don't you think? You make it sound like computer systems are notoriously unreliable, perhaps due to their inherent complexity. Do you really think the recent history of computing bears out such a pessimistic outlook? -- Drew
Post #315,627 by folkert 10/14/09 1:22:54 PM Reply	I think it was right on... I personally feel that the entire stack of people responsible for this screw up: The people that gave the OK to Hitachi. FIRST! The people in charge of backups (tested backups). The people that planned this event without properly recognizing the risks and there fore having the contingency plan all setup. The SAN Operators that allowed this to proceed with out proper SNAPSHOT saved off to another SAN, unless they were (provably) strong-armed. Last but not least the CTO and IT Managers, possibly even the CEO as the ultimate responsibility is his. And Yes, Computers are KNOWN to be that un-reliable. Google doesn't even fix broken machines in data centers. They have a policy that IF a machine stops responding, a reboot request to the "console setup" they have. If the machine comes back, it is automatically re-imaged and made "better". If it fails to respond, it is shutdown and left to decay in place until the "rack" itself is removed. They figure its going to cost them a minimum of $600 to send a warm body out there, find the machine, reboot it and (possibly) fix it. When the cost of a new machine for them is so low... its not even worth the time and effort to deal with failures any other way than to ignore them.
Post #315,628 by Another Scott 10/14/09 1:24:57 PM Reply	Well played, sir. :-)
Post #315,629 by folkert 10/14/09 1:28:19 PM Reply	Oh dude... did he zing me or what? Of course, I live it. So its hard to see the sarcasm when you are so close to it.
Post #315,632 by Andrew Grygus 10/14/09 1:53:11 PM Reply	That should be "'hanged' from the nearest yardarm" . . . . . hung is something else.
Post #315,634 by daemon 10/14/09 2:26:33 PM Reply	rman to a different array than database I insisted on an rman to virt but was overruled. Its gonna happen sooner or later do have georedundancy for a different application in place, those folks understood

Welcome to IWETHEY!