Piled Higher and Deeper
A long, long time ago, when I was a young man in Basic Training at Disney Barracks in Fort Knox, Ky, a recruit in Bravo Company, 2/13, I had a Drill Sergeant named Shirley Mason. He was a light skinned black man with freckles, reddish hair, and a hairlip, so when he spoke it was with a pronounced lisp. Sometimes it was all we could do to keep from laughing when he yelled at us-
“YOU PRIVATETHHH ITHHH ALL FUCKED UP. YOU ITHH MORE FUCKED UP THAN A THOUP (soup) THANDWICH (sandwich)”
But we didn’t laugh because he was 5’5″ of coiled muscle and anger and we were terrified of him and deified him and loved him all at once.
One day we were in the middle of doing something when it was allowed for us to be talking amongst ourselves while. Unbeknownst to my little crew of four to five guys, Drill Sgt. Mason had been in earshot of us, and had heard us bitching about all the stuff we had to do. It was the usual stuff “I hope we get off early” or “I hope we don’t have to do this” and on and on. Right after I had told Chris Plandell and Brian Carney, two other privates, that I wished we didn’t have to do our weekly fitness test in the morning, Drill Sgt. Mason hollered out in his booming voice:
“Private Cole, I want you to withhhh (wish) in one hand and thhit (shit) in the other and you tell me which one fillth up firthht.”
We received another email update today from David at Bizbudding, which I will share with you in its entirety:
“Hi John
I had a conversation today with two senior members of the 365 team.
They have advised me that there is quicker movement and forward progress and are hopeful to provide us with a timeline for the restoration of your website(s) this week.
Based on 365’s discussions with their cyber forensic partners and their current estimates, they feel this timeline is appropriate to bring the cloud data center back into service.
Again, they confirmed their assessment that customer data is secure and untouched.
365 committed to providing a post-mortem detailed root cause analysis (after all their customers have been restored) that explains the situation and why it is taking so long to restore the environment.
I will share that report with you when I receive it. I will also provide additional updates when received.
-David”
I immediately called him after receiving it, and he and I (and, as I just checked my phone before writing this, Watergirl as well) think it is more boilerplate bullshit and nondisclosure from 365, but it IS the first time since this whole nightmare began that they actually stated that the site will be coming up, even if they did not give us a timeline. Bringing it up “this week” is the first time they have even given us an estimate, however abstract and non concrete it might be.
We know how Drill Sergeant Mason felt about hoping and wishing, and I pretty much feel the same way as a prematurely cranky 50 some year old, but right now hoping and wishing is all we really have. So that is how I am looking at things. All we need is access to our data, and we will be back. I hope it is this week, but who knows. Until then, we just soldier on.
In other news, I have seen no appreciable response to my attempts to shame 365 Data Centers in my twitter campaign against them, although I will note that Stephen Klenert, Senior VP of Customer Solutions who was posting tiktoks of him cleaning his patio has now made his account private, so there is that. The whole twitter campaign probably won’t do anything other than make me feel better, but that’s enough for me to keep going.
Other than that, I got nothing.
Be good to each other.
John
Cole’s writing is the best
>>Be good to each other
Why start now?
Thanks for the update. Appreciate the communication (See how that works 365?) and sorry you have to deal with this frustration.
Thanks for the update but nope. Fucking ridiculous garbage in.
Poor you. Poor us. Fuck them.
Let us know what we can do to help us move on.
WaterGirl translates the word salad.
(This is our site host speaking for himself, not passing along 355 info.) I had a conversation today with two senior members of the 365 team.
Someone finally fucking called me.
They have advised me that there is quicker movement and forward progress and are hopeful to provide us with a timeline for the restoration of your website(s) this week.
They hope to maybe be able to tell you sometime this week when they might actually restore the websites. They thought “quicker movement” and “forward progress” would make you feel better, even though they don’t actually modify any action in particular.
Based on 365’s discussions with their cyber forensic partners and their current estimates, they feel this timeline is appropriate to bring the cloud data center back into service.
355 talked to their outside cyber folks and they are all patting themselves on the back because their timeline is appropriate! 365: We are doing a good job!
Again, they confirmed their assessment that customer data is secure and untouched.
This was an actual statement with a noun and a verb and it actually conveys meaning.
365 committed to providing a post-mortem detailed root cause analysis (after all their customers have been restored) that explains the situation and why it is taking so long to restore the environment.
355 is waiting to tell you the cause, and they are going to delay this as long as possible in the hopes that you will have forgotten the pain of this whole experience and maybe you won’t notice if they never tell you anything.
I will share that report with you when I receive it. I will also provide additional updates when received.
(This is our site host speaking for himself, not passing along 355 info.) I can’t tell you anything because they aren’t fucking telling me anything. But when they do, I will tell you.
Good on you brother. What a perfect analogy.
@Baud
If you don’t start now Bill & Ted will land an antiquated phone booth on you while still failing to actually play a guitar.
So something weird.
A high school classmate of mine (not really a friend of mine, but friends of friends) died of Covid last summer, age 41. It was during the Delta wave, so I presume he was unvaccinated. Left behind a wife and two young kids.
Anyway, so a friend of Mr. Suzanne’s (again, not really a friend, more of a friend of friends) just announced that he is engaged to my classmate’s widow.
This is some weird shit.
They did not give you a timeline. They said they would give you a timeline in a week. That language is very specific…and very misleading.
@The Moar You Know
That was my read, as well. (see my translation above)
If I recall the website went down right about the time Pentagon officials were testifying to congress about UFO’s.
Coincidence?
I posted a response in the other thread, but Watergirl is right on the money here.
They’re not committing to fixing the problem this week, only that they think they can tell you when things come back online. They could still get you (or at least some customers) up and running soon, but they won’t commit publicly to it in a way that might be actionable if they’re wrong.
I’m guessing ransomware or a big hack that compromised access tokens and 2FA, to the point where they have to redo EVERYTHING. Most data centers have enormous liability if they’re down for even short periods of time (10+ hours) so they’re really in trouble here.
Also, don’t expect much from the public post mortem. Since they would predictably be supporting evidence in a lawsuit, public post mortems on events of this scale are usually VERY low on specifics or admitting error.
Hi, WG. Any chance of getting On the Road going here? Maybe not the submission page, but posting something sent via emai?
@Origuy
Yeah, we don’t have a form. But people could send me pics and text.
But there is a 2 MB limit for the size of each photo, so people would need to be mindful of that.
But I’m willing, if there’s interest.
“Most data centers have enormous liability if they’re down for even short periods of time (10+ hours)”
Less. The SLA adjustments will bankrupt this company.
A long time ago I was in some technical class or other and someone was relaying a story of trying to debug some weird problem that crashed a very high-end switch at a data center belonging to a financial exchange, and someone in ops there said “take all the time you need, 10 minutes, 15…”
In an iota of news from the good side, electronic item ordered from Amazon during the wee hours of Friday morning here, which item upon ordering was tagged “Expected arrival May 27 to June 3” discovered in my mailbox … today.
Stunned, I iz.
MORE FUCKED UP THAN A SOUP SANDWICH
Hahaha, I hope I can use this sometime.
@Suzanne
Damn. That is weird! And god only 41. I’m not really not that far away from 41. The flu, COVID is not.
I feel bad for the kids who have to grow up without their father
All I can say is I sure hope the old-new site will be there for the January 06 hearings, becuz if it ain’t, heLL wiLL bE PaID!
😉
@Goku
You aren’t that far from 31. 🙂
You know that I was in Disney Barracks in AIT in 1967?
@GinAndTonic It depends on what the SLA is. but usually they promise more than 99.99% uptime every quarter and no more downtime than X minutes or hours in a row. How bad damages are depends on what their SLA actually is.
This CAN ruin a company easily, but if they can show they took all reasonable precautions and still got owned then they can limit exposure. That’s why I expect very little in thr post mortem and that they won’t admit to anything.
Hey John, I want to let you know that I am so fucking glad this site went down because it made be realize how much I rely on Balloon Juice to be my unofficial support system, without anyone having to do anything other than be themselves. I don’t have to comment or do anything beyond reading each day to know that I belong. And the fact that you, WaterGirl, and others, are trying so fucking hard to get this thing back up and available is so awesome – thanks to all of you.
I was kind of hoping for his and your sake that Stephen Klenert wasn’t Senior Veep of Customer Solutions. Idjit he is.
Again, they confirmed their assessment that customer data is secure and untouched.
This was an actual statement with a noun and a verb and it actually conveys meaning.”
My translation: We hope and pray that your data is safe. We have no reason to believe it is or is not, but we are hoping for the best It is secure, so damn secure that neither you nor I can touch it..
@G&T: It might depend on whether they can legitimately make an argument that the incident is covered by a force majeure clause.
@MisterForkbeard – I’ve signed these kinds of contracts, and have enforced them. Down is down. It’s just a question of whether the company has insurance and can make their insurer pay.
>>>355 is waiting to tell you the cause, and they are going to delay this as long as possible in the hopes that you will have forgotten the pain of this whole experience and maybe you won’t notice if they never tell you anything. <<<
The NFL never did tell us the results of measuring the air pressure in the footballs for an entire season, did they.
@different-church-lady
“The NFL never did tell us the results of measuring the air pressure in the footballs for an entire season, did they.”
The very quintessence of a deflationary spiral.
😉
@watergirl I can bump the file size limit to 10 megs or so if you like.
“Private Cole, I want you to withhhh (wish) in one hand and thhit (shit) in the other and you tell me which one fillth up firthht.”
God, I’m dyin’ here laughing so hard. Grade-A blogging, Cole. Sorry about the convo with 365. They’ve clearly been fucked with a ransomware attack and have no idea what to do.
Orioles beat the Yankees!
Go Blue Jays!
Go Padres!
@mrmoshpotato
https://www.youtube.com/watch?v=NR6UiSNHT3w
Sorry for the naked link
Have not read the comments so if someone already said this I apologize.
With decades of IT consulting under my belt.. that statement says they wiil give you an estimate this week .. not the website will come up this week.
Hope I’m wrong but that’s how I read it.
@kmax
Should have read the comments first 🙂
There are a couple of things that are still mysterious to me in this story.
One mystery is this: OK, so BJ is just a noisy political blog. But 3xx (good one, G&T) has real customers who pay real money for real services encumbered by real legal liabilities. I believe I have seen hospital systems mentioned. It would certainly not be surprising to find school registrars or law firms among their customers. So, why is everyone keeping quiet? How is it that there is no squawking about *truly* essential services going dark? Where are the news stories?
Something is not right here. An incentive is at work that we aren’t seeing. Probably legal, maybe financial too. But what is it? I have no clue. But there’s definitely an anomaly, requiring an explanation. If a hospital system has its patient records go offline for 10 days and that’s not a news story, WTF?
The other thing is, if I’m having a conversation with these idiots, and they say the words “we have verified that we have *backups* of your data”, then my next words to them are “I require you to turn over an SSD containing a copy of that backup within 48 hours” and follow that up within the hour with a letter from an attorney reiterating the demand. Come on, WTF? The bullshit with their investigation is A TOTALLY SEPARATE ISSUE. If they have a fucking backup, they can fork it the fuck over. You can drop the site file and the database into a WordPress instance at a different provider, do a bit of DNS magic, wait for everyone’s DNS cache to clear, and voila! Instant Reborn Balloon Juice. There is no fucking reason at all to keep our wagon hitched to this goddamn clown caravan.
@Goku
Hehe, go O’s!
What BAM said.
@Carlo Graziani
The thing is that 355 is a cloud data center provider. We aren’t their customers. The hospitals aren’t their customers. The fortune 500 companies aren’t their customers.
Their customers are the site hosts, like our site hosting company and other site hosting companies.
So Balloon Juice, and the other monetized blogs, and yes, the hospitals and the fortune 500 companies are paying real money to the site hosts.
The site hosts in turn pay real money too 355.
I am the last person to defend 355, but until 355 understood exactly what all had been compromised, it would have been irresponsible of them to turn over anything until/unless they understood what had happened.
I think they announced on Saturday that they “now know the root cause of the problem”, and that’s when they started making noises about “recovery”.
That’s when it got hinky, in my opinion. Because once you know the source of the problem and you’re talking about recovery, then we should be seeing action instead of hearing word salad.
@ Carlo G
That is pretty much what I have been thinking.. how can there not be an uproar over this?
It just does not seem right to me.
https://www.news4jax.com/news/local/2022/05/23/vystar-credit-unions-online-mobile-banking-platforms-remain-offline-more-than-a-week-after-outage-began/
As VyStar Credit Union’s online and mobile banking issues continue to frustrate members more than a week after the problems began, the Florida Office of Financial Regulation said it is monitoring the situation.
@WaterGirl: I do have some notion of the B2B clusterfuck. And I know that you have a business relationship with your site host that you value. But. That degree of separation is what is preventing you from putting the screws on 3xx.
I used to have a similar sort of bullshit 3-way relationship with an ISP and AT&T, back when it was possible to have multiple service providers with a single data (DSL, at the time) provider, so at the individual level I know how annoying and frustrating it is to deal with nice, friendly people who provide the layer of isolation from the infrastructure morons who casually fuck everything up.
That relationship is the source of your problem now, in my opinion. It doesn’t matter how nice those guys are. Either *they* demand those backups for you, on your behalf, using their legal staff, or you put the legal clamps on them. That’s the way it works in the US. 3xx has your backups. Technically, BJ could be back up and running in a couple of days, if they cooperated. You need to get 3xx to disgorge those backups. If the path to do that is to make a legal threat against your provider, so that they, in turn make one against 3xx, then, why not do that?
Jackal Legal, please chime in.
And to think I beat myself up over like accidentally taking an app down for an hour.
The guy who deleted prod last year is still unrepentant, though. Maybe I’m the weird one.
Reddit thread from someone who claims to be on the inside:
https://www.reddit.com/r/sysadmin/comments/urybn4/365_data_centers_ransomware_attack/i9rg0f8/
I have no idea if it’s legit. Caveat emptor.
FWIW.
Cheers,
Scott.
Full thead: https://www.reddit.com/r/sysadmin/comments/urybn4/365_data_centers_ransomware_attack/
The tweet about 355 Data Centers is utterly unfair. They are just prorating 50 years worth of leap-days.
gotta be ransom. even simple RAID strategies from the late 80’s would have been recovered in a day or two; the bulk of which would be getting new disks shipped.
I suppose a total site loss (like a tornado took the building) if they don’t have an offsite disaster recovery plan but that’s pretty incompetent too.
Might have been posted already, and might not have any new info, but here’s a Reddit thread with comments from someone claiming to be an employee suggesting that the company is fucked no matter when the data is returned to clients.
https://www.reddit.com/r/sysadmin/comments/urybn4/365_data_centers_ransomware_attack/?utm_source=share&utm_medium=ios_app&utm_name=iossmf
This week they might give you a timeline, I see no commitment to getting you up and running this week.