How the Romney GOTV effort died

Post #366,059 by malraux 11/9/12 6:39:43 PM Reply	How the Romney GOTV effort died Yes, it was a software problem. http://arstechnica.c...f-an-it-meltdown/ To build Orca, the Romney campaign turned to Microsoft and an unnamed application consulting firm. The goal was to put a mobile application in the hands of 37,000 volunteers in swing states, who would station themselves at the polls and track the arrival of known Romney supporters. The information would be monitored by more than 800 volunteers back at Romney's Boston Garden campaign headquarters via a Web-based management console, and it would be used to push out more calls throughout the day to pro-Romney voters who hadn't yet shown up at the polls. A backup voice response system would allow local poll volunteers to call in information from the field if they couldn't access the Web. But Orca turned out to be toothless, thanks to a series of deployment blunders and network and system failures. While the system was stress-tested using automated testing tools, users received little or no advance training on the system. Crucially, there was no dry run to test how Orca would perform over the public Internet. Part of the issue was Orca's architecture. While 11 backend database servers had been provisioned for the systemÂprobably running on virtual machinesÂthe "mobile" piece of Orca was a Web application supported by a single Web server and a single application server. Rather than a set of servers in the cloud, "I believe all the servers were in Boston at the Garden or a data center nearby," wrote Hans Dittuobo, a Romney volunteer at Boston Garden, to Ars by e-mail. Throughout the day, the Orca Web page was repeatedly inaccessible. It remains unclear whether the issue was server load or a lack of available bandwidth, but the result was the same: Orca had not been tested under real-world conditions and repeatedly failed when it was needed the most. By way of contrast, Obama's GOTV application, Narwhal, was hosted on AWS and developed using open source tools. Regards, -scott Welcome to Rivendell, Mr. Anderson.
Post #366,061 by Another Scott 11/9/12 6:41:31 PM Reply	GMTA. #366058 http://forum.iwethey...iwt?postid=366058 Cheers, Scott.
Post #366,063 by Another Scott 11/9/12 6:57:02 PM Reply	Classic. At one point, the network connection to the Romney campaignÂs headquarters went down because Internet provider Comcast reportedly thought the traffic was caused by a denial of service attack. Sheesh. Cheers, Scott.
Post #366,066 by drook 11/9/12 7:35:41 PM Reply	That's interesting Some apps you just know are going to go from zero traffic to holy-shit-that's-a-lot-of-traffic traffic in the span of a day. I wonder how a provider can be prepared for that. -- Drew
Post #366,067 by rcareaga 11/9/12 7:40:03 PM Reply	asked and answered The reliably contemptible "Powerline" blog carried a brief description of Orca on Monday. Emphasis added, because I'm feeling helpful: The Democrats like to think that their Âground gameÂ is the tie-breaker that will give them the edge in a tight election, but this year there is reason to believe that the DemocratsÂ efforts will be equaled, if not exceeded, by those of the Romney campaign. The Romney campaign is not only well-funded, but is run by one of the best organizers and managers of his generation, the candidate himself. Whom would you count on to organize anything, Mitt Romney or David Axelrod? http://www.powerline.../project-orca.php cordially,
Post #366,099 by Another Scott 11/10/12 5:39:10 PM Reply	Good comments. This strikes me as making sense, but I have no way of knowing if it's correct - http://arstechnica.c...#comment-23482379 It's pretty obvious what happened. The code was late and didn't work very well. They were still working on it at least the night before the election. That's why the PDFs with account info were so late. They had just been auto-created and emailed. 11 database servers seems pretty crazy, but I suppose that could've been a good idea. However, you better believe that the single web/app server was not planned. Again, project running late, and they were likely having problems with load balancers and application server clustering. There were likely some critical bugs that made it possible to only run a single app server. Nonetheless, that's just one of many problems on the project. They completely failed at training, infrastructure engineering, and software design. Pretty much what I would expect from consultants working on a lucrative campaign project. Last edited by jbrown96 on Fri Nov 09, 2012 4:54 pm Presumably the sensible people at the RNC (those who actually try to run the thing in the trenches) are trying to figure out what went wrong so it doesn't happen again in 2016. A good argument for vote-by-mail - throw out the need for Narwhals and Orcas. ;-) Cheers, Scott.
Post #366,100 by malraux 11/10/12 5:55:41 PM Reply	Redstate in a lather http://www.redstate....nsultant-con-job/ They say that the truth is the consultants essentially used the Romney campaign as a money making scheme, forcing employees to spin false data as truth in order to paint a rosy picture of a successful campaign as a form of job security. Zac Moffatt, Digital Director for the Romney campaign, was specifically named as having Âbuilt a nest egg for himself and co-founder of Targeted Victory, Mike Beach,Â and that they ÂdidnÂt get socialÂ media and ignored objections from other consultants and staffers in the campaign. Here's the thing: no one should be amazed that Romney can't run a company or manage consultants. He's a financial guy, and a private equity financial guy at that. He's never actually run a company that builds something, makes something, or provides a service. All of his vaunted Bidniss Experience was useless in the real world of doing things. Private equity people can sometimes be good at this, but not if they haven't come from another business that actually had to do something for a bottom line. Invariably the inside guys, the deal makers, the ones who know money inside and out but nothing else, are complete and utter idiots when it comes to the nitty gritty of running a company. They're the guys on the board that make a vassal company's CEO or CIO cringe with every dumb-ass idea they have for increasing revenue. Romney also hasn't had to deal with information technology on a large, enterprise scale. Bain was small. They didn't need a huge IT infrastructure, and as a result he never understood it. Some financial companies, like my old one, had to be able to handle this sort of thing because they were actually providing a service and managing high volume trading systems. Not Bain. Regards, -scott Welcome to Rivendell, Mr. Anderson.
Post #366,103 by Another Scott 11/10/12 7:07:07 PM Reply	Yup. But Romney was worse than that. He, somehow, managed to have the federal government bail out his enterprises on multiple occasions. 1) The Salt Lake City Olympics was rescued by the US taxpayer - http://www.mediaite....ic-welfare-queen/ - not by some magic managment skills on his part. 2) Romney supposedly screwed over the FDIC to save Bain in 1990s and made millions in the process - http://www.rollingst...t-romney-20120829 Few expect Romney to be up on the ins-and-outs of running a big IT project, but this Orca episode is yet another illustration of his incompetence in running a campaign. And his campaign seems to fit with the reality of his professional career - his "I'm the CEO so pay fealty to me" attitude and unwillingness to accept criticism (recall his attitude about being questioned during the "corporations are people my friend" episode), his history of bullying in school, his "service" in Paris during Vietnam, his kids "service" in working on his campaign, etc., etc. I think he lost because a majority of people didn't like his policy proposals and his associations with reactionaries in Congress, his lies, his refusal to have press conferences, his refusal to release his tax returns, and so forth, but his incompetence in running his campaign didn't help. Good riddance. My $0.02. Cheers, Scott.
Post #366,106 by boxley 11/10/12 8:12:01 PM Reply	repos are in agreement unseating a sitting president is difficult so all of the whack jobs numbnuts and romney were encouraged to run. Romney won because they wanted him to just go away. Next time , he wont be invited. As a side result the "campaign" was actually a huge bust out operation run under the financial titans noses as they grovelled and tipped their caps while stealing the place blind. Any opinions expressed by me are mine alone, posted from my home computer, on my own time as a free American and do not reflect the opinions of any person or company that I have had professional relations with in the past 55 years. meep
Post #366,163 by mmoffitt 11/12/12 8:33:31 AM Reply	They certainly filled the bowl. But it remains to be seen whether or not they'll push the handle down. When people like Boehner, Coburn and Graham are the "sane" members of your party, you can't afford to flush all the crazies because nobody will be left.
Post #366,132 by drook 11/11/12 10:32:45 AM Reply	Says more about otusourcing than about Romney specifically Those two lines you quoted could be applied to nearly every major outsourcing project I've heard of. I've read job descriptions for consultants that very prominently -- and very openly -- describe as one of the chief responsibilities "increasing billables and recommending additional services". -- Drew
Post #366,258 by Ashton 11/14/12 4:47:39 PM Reply	Your summary of Rmoney.. (Man and/As Machine) is the most concise and maybe the most simply-accurate ... I've read. He wore his (obvious) general insouciance as a Badge of Repo Honor. So glad! it stabbed him with curare. Cosmic humour all around. Nicely said. :-)
Post #366,363 by rcareaga 11/16/12 2:35:20 PM Reply	Narwhal vs Orca Interesting counterpoint to "what Romney's consultants did wrong": what Obama's tech people (not consultants) did right. Longish article, but well worth the read. The Obama campaign's technologists were tense and tired. It was game day and everything was going wrong. Josh Thayer, the lead engineer of Narwhal, had just been informed that they'd lost another one of the services powering their software. That was bad: Narwhal was the code name for the data platform that underpinned the campaign and let it track voters and volunteers. If it broke, so would everything else. They were talking with people at Amazon Web Services, but all they knew was that they had packet loss. Earlier that day, they lost their databases, their East Coast servers, and their memcache clusters. Thayer was ready to kill Nick Hatch, a DevOps engineer who was the official bearer of bad news. Another of their vendors, StallionDB, was fixing databases, but needed to rebuild the replicas. It was going to take time, Hatch said. They didn't have time. They'd been working 14-hour days, six or seven days a week, trying to reelect the president, and now everything had been broken at just the wrong time. It was like someone had written a Murphy's Law algorithm and deployed it at scale. And that was the point. "Game day" was October 21. The election was still 17 days away... http://www.theatlant...rching-in/265325/ cordially,
Post #366,365 by boxley 11/16/12 3:00:07 PM Reply	good find Any opinions expressed by me are mine alone, posted from my home computer, on my own time as a free American and do not reflect the opinions of any person or company that I have had professional relations with in the past 57 years. meep
Post #366,373 by Another Scott 11/16/12 4:23:34 PM Reply	A very good read. Thanks.
Post #366,555 by crazy 11/20/12 7:04:57 AM Reply	Great testing When I was wee tech lad, and had about 2 months of Xenix use/self training/customer tech calls, my boss decided it was time to test me. He was a Unix admin, classic nerd, with deep AT&T and Berkeley connections. He went through and fucked up the system. He changed permissions, contents of driver files, removed critical files, renamed them, it was ALL OVER THE PLACE. And then he said: Fix it.
Post #366,565 by folkert 11/20/12 10:51:57 AM Reply	Hahaha... I accidentally did that kinda stuff to myself on a production machine... accidentally. I was CLEARLY in a testing environment on the OLD HPUX v8.02.something HP 817 machine. I thought I was in root's homedir at the time. (whoops) In this particular machine with 8.02.... "/" was the homedir for "root"... and it wasn't 8.02... was 9.04f and on the new(er) box... CRAP! Ummm yeah and its wasn't the old test machine. Whoops... all printing, data entry, workings... down while I called HP for support. Surprisingly, HP had a script (on the machine already) to fix this. Seems I wasn't the only one to have done this in the past. -- greg@gregfolkert.net PGP key 1024D/B524687C 2003-08-05 Fingerprint: E1D3 E3D7 5850 957E FED0 2B3A ED66 6971 B524 687C
Post #366,556 by crazy 11/20/12 7:25:06 AM Reply	They should have asked me An Operational Data Store (ODS, a backend chunk of data that needs to be accessed in real time by real people) is NOT a Data Warehouse. It directly conflicts with it. We knew that back in the 90s. While the dream had been for all applications to run through Narwhal in real time, it turned out that couldn't work. So, they split things into real-time applications like the Call Tool or things on the web. And then they provided a separate way for the Analytics people, who had very specific needs, to get the data in a different form. Then, whatever they came up with was fed back into Narwhal.

Welcome to IWETHEY!