Strange goings on in the cloud

Post #441,492 by Another Scott 5/17/22 8:14:26 AM 5/17/22 8:14:26 AM Reply	Strange goings on in the cloud https://Balloon-Juice.com has been unreachable for days. There's a message up saying that it's a problem with the cloud provider. Things like this are never supposed to happen. The fact that it is taking days (rather than minutes) to work around is not a good sign for that company... :-( I wonder if this is going to end up being a case study of some sort. Cheers, Scott.
Post #441,494 by scoenye 5/17/22 11:23:19 PM 5/17/22 11:23:19 PM Reply	The stack of case studies must be touching the ceiling by now As more and more eggs end up in the same basket, the scope and impact of even small incidents gets ever worse. The complexity of the interconnections is such that no one has a complete grasp of it anymore. Atlassian took itself and its customers offline for ~2 weeks last month following an accident with a maintenance script. The potential intrusion is bad, but I can't really fault them for taking the time to make sure everything is secure before releasing anything for use. Given the geopolitical situation and the damage potential of current malware, they really didn't have much of a choice.
Post #441,495 by CRConrad 5/18/22 2:05:40 AM 5/18/22 2:05:40 AM Reply	The strange thing is that everything IS in the fricking "cloud" in the first place. Did ALL the senior manglement who heaved those giant sighs of relief at the escape from centralised monopolist IBM back in the 1980s die out already? Did NONE of their successors, even if they weren't personally around for that, learn the lesson from them? -- Christian R. Conrad The Man Who Apparently Still Knows Fucking Everything Mail: Same username as at the top left of this post, at iki.fi
Post #441,496 by drook 5/18/22 9:11:19 AM 5/18/22 9:11:19 AM Reply	One thing hasn't changed yet Economies of scale are still real, so resources keep getting bigger, and (usually) shared by more people. Which means when there's a problem it affects more people. And when it's a security issue, just identifying the potential scope requires searching a much larger surface. Real question: What lesson should they have learned in the 80s that would have prevented this? -- Drew
Post #441,497 by boxley 5/18/22 10:39:04 AM 5/18/22 10:39:04 AM Reply	data is not math "Science is the belief in the ignorance of the experts" – Richard Feynman
Post #441,498 by drook 5/18/22 12:27:28 PM 5/18/22 12:27:28 PM Reply	And what would they have done differently if they'd learned that? -- Drew
Post #441,500 by boxley 5/18/22 1:15:35 PM 5/18/22 1:15:35 PM Reply	quit using spreadsheets as a database "Science is the belief in the ignorance of the experts" – Richard Feynman
Post #441,501 by drook 5/18/22 3:26:17 PM 5/18/22 3:26:17 PM Reply	How does that prevent an outage when your host shuts down to respond to an attack? -- Drew
Post #441,503 by boxley 5/18/22 5:27:56 PM 5/18/22 5:27:56 PM Reply	it doesn't "Science is the belief in the ignorance of the experts" – Richard Feynman
Post #441,504 by CRConrad 5/19/22 2:18:01 AM 5/19/22 2:18:01 AM Reply	That not putting all your eggs in someone else's single basket beats economies of scale, IMO.
Post #441,506 by malraux 5/19/22 10:15:33 AM 5/19/22 10:15:33 AM Reply	In my experience that cloud basket is better than a company's single basket Especially with availability zones and multiple regions. Most companies don't have the capital or expertise to set up a local equivalent that even remotely approaches the robustness of a big cloud provider. The enormous cost savings of not owning multiple physical plants (you need at least two, separated far enough geographically that they're on separate electrical grids and in different natural disaster regions) can be spent on hardening your cloud presence, with plenty of money left over for other things. If you're truly concerned about a single provider, use Terraform or Serverless to spread your infrastructure across multiple providers. Complexity goes way up, however, and the incremental benefit over using multiple regions in a single provider probably isn't worth the investment. As an example, Google's SRE discipline is aimed at hitting 99.99% reliability. Anything over that isn't going to be noticed by users and the incremental improvements are cost-scaled way out of proportion to the improvements. Companies I've worked for have had issues from 3rd party SaaS vendors way more often than the IaaS cloud providers. In the past 8 years I can think of 2 times where we were directly affected by a cloud provider's issues, and maybe 2 or 3 times indirectly. Having said that, avoid AWS' us-east-1 as most of the issues seem to happen in that region. Regards, -scott Welcome to Rivendell, Mr. Anderson.
Post #441,510 by drook 5/19/22 11:32:02 PM 5/19/22 11:32:02 PM Reply	Pretty sure we're using us-east-1 Got anything you can point to showing why we shouldn't? -- Drew
Post #441,511 by malraux 5/20/22 9:32:56 AM 5/20/22 9:32:56 AM Reply	Most of the big failures I've seen have been there Like the big December '21 outage: https://www.zdnet.com/article/aws-heres-what-went-wrong-in-our-big-cloud-computing-outage/ Regards, -scott Welcome to Rivendell, Mr. Anderson.
Post #441,514 by drook 5/20/22 6:27:33 PM 5/20/22 6:27:33 PM Reply	Thanks, got some people I'll forward this to -- Drew
Post #441,517 by malraux 5/21/22 11:23:14 AM 5/21/22 11:23:14 AM Reply	There's a 3rd party vendor we recently rejected Their availability averaged 96% over the past 6 months, and a number of the outages and degradations were either caused by or exacerbated by AWS issues, all in us-east-1. Regards, -scott Welcome to Rivendell, Mr. Anderson.
Post #441,521 by crazy 5/21/22 11:32:17 PM 5/21/22 11:32:17 PM Reply	2 weeks of downtime per year That's really bad.
Post #441,520 by static 5/21/22 10:37:57 PM 5/21/22 10:37:57 PM Reply	I think us-east-1 is their original location. So there must be some hardware build decisions left over from when it was first created.
Post #441,526 by malraux 5/22/22 3:11:58 PM 5/22/22 3:11:58 PM Reply	Yes, there are a number of oddities with that region. They finally removed the weird "S3 buckets are global, but really they're just in us-east-1" problem, but there are still a number of global API endpoints that are hosted there. This can make locking down regions that can create new resources difficult if one of those regions isn't us-east-1. Regards, -scott Welcome to Rivendell, Mr. Anderson.
Post #441,499 by crazy 5/18/22 1:03:48 PM 5/18/22 1:03:48 PM Reply	The first time I coded for the web it was 3270 all the way down Handling the browser portion in forms that contained multiple fields and slinging information back and forth to the back end which I also programmed. I commented to the mainframers that this is just like a 3270 terminal working in a mainframe interface. Sure you got fonts and graphics but internally it's pretty much the same.
Post #441,502 by drook 5/18/22 3:42:47 PM 5/18/22 3:42:47 PM Reply	MVC as a conceptual description hasn't changed much It's still just UI, data (model), and the glue between. The only thing changing is where the code for the different pieces is running. Maybe I'm officially one of the old farts who just doesn't think the new stuff is any good, but I really haven't seen much that's genuinely new. Machine learning and distributed parallel computing are huge, but that's not what most people are working on. -- Drew
Post #441,505 by CRConrad 5/19/22 2:25:08 AM 5/19/22 2:25:08 AM Reply	Mine was kind of similar. Twenty years ago, when I was a DW consultant at Oracle. A part of the reporting system for a client, kind of a "two-level recursive meta" thing: PL/SQL that generated HTML/JS that, when clicking a link or button to drill down, called other HTML/JS generated by other PL/SQL procedures. Having built similar stuff in Delphi at my previous job, it felt pretty damn stone-age. (And "Visual" Source Safe didn't make that better.) -- Christian R. Conrad The Man Who Apparently Still Knows Fucking Everything Mail: Same username as at the top left of this post, at iki.fi
Post #441,524 by Another Scott 5/22/22 2:42:13 PM 5/22/22 2:42:13 PM Reply	Cole brings out the flamethrower. Thread: https://twitter.com/Johngcole/status/1528441553043193862 Cheers, Scott.
Post #441,549 by Another Scott 5/24/22 12:58:53 AM 5/24/22 12:58:53 AM Reply	Reddit thread claims that it's ransomware https://www.reddit.com/r/sysadmin/comments/urybn4/comment/i9rg0f8/ No idea if it's legit. Cheers, Scott.
Post #441,582 by rcareaga 5/26/22 10:06:34 AM 5/26/22 10:06:34 AM Reply	Scott’s probably already seen this A comment posted in this thread at the Balloon Juice home-away-from-home refugee encampment, in which commentator “Lacuna Synecdoche” translates a communiqué from corporate weaselese: …industry experience and cynicism make me suspect that the translation from business-speak/legalese is more like this: 365 DC: “…evaluation to date by our systems team and cybersecurity experts has revealed that, aside from the targeted third party, no data was taken from the 365 Data Centers cloud environment …” Translation: “We think they encrypted everything as quickly as they could access it, meaning they probably didn’t scan for credit card numbers and personal information. Yet.” 365 DC: “…365 Data Centers believes that at this point in time the prudent path forward is to rebuild the affected cloud platform.” Translation: “You’ll need to sue us to get your money and data back.” 365 DC: “This will be conducted along with an all-out effort to retrieve all data within the existing cloud environment that can still be accessed.” Translation: “We have some backups from 2019. If any of your files are among them, we’ll give them back to you in exchange for a settlement where you indemnify us against any other damages. We don’t have anything more recent than that, because we stopped doing any due diligence or maintenance in 2020, figuring we could blame any problems on Covid.” 365 DC: “If your preference is to work with us to restore your service on a new 365 platform, please inform Steve Oakie, 365 Data Centers’ Chief Revenue Officer.” Translation: “Sure, we’ll be happy to bilk you for more money if you’re dumb enough to keep doing business with us.” 365 DC: “We are saddened by the impact this incident has caused on our many years of collaborative hard work with you to build your cloud services. Our entire organization is sorry for the significant inconvenience that this has brought to you and your business.” Translation: “You’re on your own. We’re too busy working out how to monetize our remaining assets and distribute the proceeds among management, in the form of golden parachutes, before we declare bankruptcy and dissolve the company. But, really, we do feel bad about it.” Sounds right. cordially,
Post #441,590 by Another Scott 5/29/22 10:50:36 PM 5/29/22 10:50:36 PM Reply	The site is back, but no backups yet. https://www.balloon-juice.com 15-ish days, and still nothing useful from 3xxDC. :-/ A class-action lawsuit has been filed - https://www.courtlistener.com/docket/63345824/bizbudding-inc-v-365-data-centers-llc/ 72-page complaint - https://storage.courtlistener.com/recap/gov.uscourts.ctd.149894/gov.uscourts.ctd.149894.1.0.pdf Cheers, Scott.
Post #441,592 by CRConrad 5/30/22 2:46:32 AM 5/30/22 2:46:32 AM Reply	"15-ish days, and still nothing useful from 3xxDC" -- so they're down to about 350DC by now?
Post #441,593 by pwhysall 5/30/22 8:24:44 AM 5/30/22 8:24:44 AM Reply	Brutal! Plaintiffs Bizbudding Inc, (“Biz”), Parisi Speed School (“Speed School”), Core Wellness, LLC (“Core Wellness”) and PaleoMom.com (“PaleoMom”) (hereinafter “Plaintiffs”) bring this class action against 365 Data Centers Services, LLC (hereinafter alternatively “365” or “Defendant”) for negligence, breach of contract, unjust enrichment, and a violation of the Connecticut Consumer Unfair Trade Practices Act (“CUTPA”), Conn. Gen. Stat. § 42-110A, et seq., based upon Defendant’s failure to secure its systems and data from cyberattacks, including ransomware attacks, failure to properly secure and manage backup data for its clients and their customers, and failure to properly segment its data security systems.
Post #441,594 by drook 5/30/22 8:52:30 AM 5/30/22 8:52:30 AM Reply	Hold a host accountable for what they should be doing? What a concept. -- Drew
Post #441,600 by crazy 5/31/22 3:20:28 AM 5/31/22 3:20:29 AM Reply	It will never happen I'm sure they've got a dozen get out of jail free cards in their contract. And you can never secure well enough to guarantee someone can't break in. Then it's a matter of: Did you do enough compared to "best practices"? And the standards are worthless. I sat through many security reviews when I was sure our systems were full of holes and the guy in charge of the security would smile at the auditor. The auditor would point out the security hole. The system administrator would pick up a best practices manual and point to the page where he fulfilled the requirement. And then they went on to the next item. Again and again. "Best practices" hides a lot of failure.

Welcome to IWETHEY!