Must resist the urge to add that image as the banner for our little home away from home.
I saw on John’s twitter feed that the Senior VP of Fucking Customer Solutions at 355 Data Centers made his TikTok private after John posted the TikTok of the guy washing down his driveway this weekend. So someone from “355” noticed John’s rants, at least.
Apparently at 355, they all have their fingers in their collective ears, chanting “la-la-la I can’t hear you.”
At this point I share Gin & Tonic’s view from Friday. How does this not become an extinction-level event for “355” Data Centers?
Anyone want to speculate as to the reason for the outage?
Disgruntled employee?
State-sponsored hack?
Random hacker doing it for grins?
Ransomware?
Incompetence?
Greed?
Under-funding of safety measures?
Why is it taking so long to resolve?
Incompetence?
It’s ransomware and they were trying to get the money?
It’s ransomware and they were stalling, hoping to fix it before paying?
All 365 employees are at a 2-week retreat, except for the poor clueless bastard who has to answer the phone?
Why all the secrecy?
Government agencies involved and they required secrecy?
Outage due to some embarrassing reason they are trying to hide?
The people at the top are uncaring dicks?
Lex Luthor is running the company?
What’s your best guess for the date we actually learn anything tangible?
choose your date
What’s your best guess for when Balloon Juice is back up?
choose your date
*****
Add your best guesses in the comments. Whoever comes closest wins… something?
If you are not in the mood to speculate, you can admire the cat quilt that Quilting Fool is making me!
I have never seen a quilt before it is quilted before!
The Senior VP watering his driveway drowned the gerbil keeping things running. Their IT department is now down to a (hibernating) 17-year cicada.
@scav
Excellent way to start us off!
Mike Lindell and his cyberninja buffoons took over the data center to carry out an ‘audit’
My gut feeling is this was a ransomware hack, and they’re keeping quiet because they don’t want to admit fault. It’s a bit more speculative, but I would be surprised if they tried to restore from backup rather than pay the ransom, only to discover their backups were compromised, too. I have no clue when Balloon-Juice will be back up, but I expect it will be at a different hosting service with a gap in the archives because they weren’t backed up.
The all-purpose answer to nearly all of our problems in the 21st century:
The people at the top are uncaring dicks.
Baby formula shortages? Check.
Climate change? Check.
Loss of voting rights? Check.
Loss of bodily autonomy? Check.
Power outages? Check.
I’m probably overly paranoid, but as soon as the site went down, I thought “Uh oh, some henchman of Putie-poot’s noticed Adam’s excellent nightly Ukraine updates and nuked the site.”
I think that [n] Data Centers secretly replaced the coffee they normally serve with Folgers Crystals.
Anyone want to speculate as to the reason for the outage?
Incompetence (which lead to all other scenarios)
Why is it taking so long to resolve?
Incompetence
Why all the secrecy
Coverup
What’s your best guess for the date we actually learn anything tangible?
During discovery.
What’s your best guess for when Balloon Juice is back up?
Depends if BJ has its own local back-ups. If so, by the end of the month with a gap on a different host. If no local back-ups, then welcome to Jackal Action.
This has made me prioritize off-site personal backups (not through or involving my provider) for my own setup. Hopefully BJ can figure out a way to do that too.
I like the comment on that Reddit thread reacting to the probability that it is a ransomware attack:
“I have a friend who works there. Says that they will likely pay this week, then go chapter 11.”
Their friend said it, it was posted on the internet, so it must be true!
Easy peasy.
When will it be back up? After the first bankruptcy court appearance of course. Duh!
@Roger Moore’s suggestion makes the most sense to me. Sure smells like a ransomware attack + incompetent management.
@alang
We already have that worked out going forward. Unfortunately, our time machine is not functioning and that bastard Barack Obama will not loan us his.
@Chief Oshkosh: “The people at the top are uncaring dicks.”
Ding ding ding, we have a winner.
(I have personal experience with this phenonenon, as I suppose a lot of people here do.)
Some vaguely amusing links on 365’s web site… or at least they were, back when it was functioning…
Overcoming the Causes of Data Center Outages
https://web.archive.org/web/20211228145917/https://365datacenters.com/resources/white-papers/overcoming-causes-data-center-outages/
“The Cold, Hard Facts About Uptime”
https://web.archive.org/web/20210413110223/https://365datacenters.com/resources/white-papers/the-cold-hard-facts-about-uptime/
“How Much Downtime Can Your Organization Afford?”
(Alas, not cached.)
July 29, 2022.
We welcome the return of Balloon-Juice.
Rapture!
The one guy who knew how to keep things running retired/died/quit and the Masters of the Universe didn’t bother to train a replacement because what-could-go-wrong?
@alang
I googled “365 data centers ransomware” earlier and up came 2 things.
The first was was 365 data centers article on preventing ransomware and the second was the link to reddit where people were saying they were under attack from ransomeware.
I found that amusing in a sad sort of way.
@WaterGirl
When did Obama get a time machine? In the future?
I thought this article might be useful. It’s from 365 Data Centers blog.
https://365datacenters.com/blog/understanding-ransomware-attacks-how-cybercriminals-infiltrate-your-mission-critical-data/
Understanding Ransomeware Attacks: How Cybercriminals Infiltrate Your Mission-Critical Data?
Oct 5, 2021 — 365 Data Centers delivers timely technical support to improve the uptime, security and availability of your business’s network infrastructure.
Ha ha! Just kidding! That link does not work, along with everything else associated with 365 Data Centers. Nothing works. They should have read their blog post from 2021. 😡
Secrecy seems actually easy. If some of their customers are hospitals then they may arguably be in extreme legal peril for improper handling medical records, one of the categories of records encumbered by the most high-explosive potential in our legal system. Also, did they handle records for any Universities or Colleges or school systems? More legal jeopardy. Sell data services to any big law firms? Just go douse yourselves with gas and set yourselves on fire now, it will be quicker and less painful.
The absolute silence on their part makes me think it’s either
a) Ransomware or
b) Incompetence of such an order that they daren’t admit to it. ie on the level of
rm -r * (for old Unix types)
@WaterGirl, @Scout211
And WaterGirl got there first.
Did 365 [sic] data centers go down the same same day that clerk quit in a small New England town? Hmmm….
@Watergirl,
Wanna borrow mine?
2-3 weeks delivery, but right now it’s stuck on late spring, 1972, a big bowl of buttery mashed, topped with fiddleheads and pan fried smelt.
Our relationship with food has changed so much. Right about now, was the time of year when food stopped coming out of a can.
@Scout211
Yeah, but you included more details!
Plus, I greatly appreciated the bonus mockery. 🙂
@WaterGirl 🥰😆 admiring noises about that quilt
@Jay
Thanks for the laugh!
@mrmoshpotato
Obama used his time machine before he even took office. It’s how he placed the fake birth announcement for himself in a Hawaiian newspaper when he was born in Kenya and how he was responsible for the response to Katrina and other lowlights of the W. presidency.
I will note that it’s 3:30+ Eastern time and there is supposed to be a phone update from 355 -> our site hosts today. I’m sure this will be happening any minute.
@Tenar Arha
She just sent the photo today, so this is the first time I have gotten to see it mostly put together. I am super pleased.
I am sad that no one else seemed to appreciate my fixes to the “355” logo as much as I did.
I have said from the start this must be a ransomware attack. This was what happened to the local community college when their systems went down for several weeks a few months ago. It shut the whole school down because no one could do anything. Of course, they were as transparent about the situation as possible as soon as they figured out what happened. I do know they didn’t pay the ransom but it took them a long time to get it all going again and it really screwed up pretty much everything for everyone there.
I’m with Cole: signs point to ransomeware
Date of reliable information: 1 June
Date site back up: maybe never on this hosting service
If 365 does not have off-site offline backups from which their customers’ data can be retrieved, they deserve to go out of business
The thing that’s really odd here is that there are resources where something like this should become (fairly) public – mailing lists and forums where serious network admin and security people hang out, and major infrastructure outages are reported/tracked. I used to follow a bunch, but being now retired, I very seldom care about this. Except now I do care, and peeking into some of those same resources shows complete silence (and no, r/sysadmin is *not* one of those.)
I certainly hope your hosting provider kept off-site backups.
It doesn’t matter when Obama will get his time machine; he has it now.
Duh, its the deep state and dark maga while snorting cocaine at a certain pizza parlor’s basement.
@Enhanced Voting Techniques: “The one guy who knew how to keep things running retired/died/quit and the Masters of the Universe didn’t bother to train a replacement because what-could-go-wrong?”
If only this were a joke, rather than a common and perfectly ordinary M.O., especially when the prime reason for any company’s existence is to be squeezed for profit by the MOTU.
I’m pretty sure it was either Antifa or BLM or some of those GROOMERS. Probably using (((Jewish space lasers))), or maybe Italian satellites.
Is there a forum where day traders hang out and swap rumors? It would be very wrong to encourage anyone to start shorting 365 Data Center shares.
@Gin & Tonic
There were off-site backups. In two different locations. So our site existed in 3 locations (the actual site, plus 2 different locations in different states).
But all 3 locations were part of the 355 data center. So we have no backups other than the ones in the 355 data centers. If we did, we would have been up and running on Sunday, after giving them a chance to get it together on Saturday.
Our site host believes them when they say are site and backups are fine. He does not believe they lied to his face.
@Watergirl,
I liked them.
@Carlo Graziani
I do not know. Earlier today i was wishing I knew how to see the history of 355 stocks and what it has been doing for the last 90 days.
I’m going with ransomware, but… yeah. It’s certainly possible that something else is going on, but it’s extremely hard for me to believe that it’s hardware related (if they have three separate site backups, for example) and if it was configuration related then a competent team should be able to roll back the changes.
So this sounds like a security incident where they’ve been locked out of their own datacenters, is my best guess.
@MisterForkbeard wrote:
“So this sounds like a security incident where they’ve been locked out of their own data centers, is my best guess.”
They did admit to “not having access to some of our tools” because of the “security incident”, so i think you’re at least partly right about that.
Clearly, ex-365 DC runs (ran?) on baby formula.
They’ve “restructured their debt” and “secured additional funding” twice in the last year.
ISPs will cut service for nonpayment.
It’s kinda like ransomeware, but legal!
@WG – Then he is a more trusting soul than I am. At this point I am fully prepared to hear you have nothing.
Col Mustard, candlestick, server room.
This is my wheelhouse. They have undoubtedly been the victims of ransomware. One of the nasty things about ransomware is sometimes you can pay up and you get nothing: “you come outside and your shit is gone” as some band said once.
That happened to a major hospital chain here in San Diego; to compound the issue, they’d never tested their backups, because when they tried to restore from that, it did not work. Patient records, chemo treatment logs, payroll – all that good hospital stuff, all gone. They’re still in business but a hospital is not a datacenter; the datacenter has nothing of value once the data is gone.
@Gin & Tonic
I don’t think that is going to happen.
This is what we got from our site host late Saturday night:
“365 has initiated a security process to reset user, endpoint, and server passwords and cryptographic keys. They have also introduced additional firewall and network isolation tools to provide an additional degree of protection.
365’s goal is to relaunch our data center exactly as it was prior to the cyber event.
I am very sorry that I don’t have firm dates yet. “
99% chance it’s ransomware.
What the hell kind of data center doesn’t have air gapped backups, anyway? Don’t answer that, I’ll get depressed.
I went and checked and archive.org does seem to have my favorite posts of mine archived, which I assume extends to other posters, so a convoluted partial recovery might be possible, worst case scenario.
Somebody should contact datacenters.com, an aggregator, which says about 3xx: “365 Data Centers has a 100% uptime record with no client impacting downtime in our facilities in over ten years”
I’d hate to think that it’s ransomware and 365 doesn’t have the funds to cover it . . . go dark until they work out a bankruptcy schedule?
@WaterGirl
This is the most specific 355 has been with our site host, so I think if we were all totally fucked we would still be getting meaningless word salad.
To those wondering about stock price, etc., 3xx Data Centers is privately held.
@geg6: “I have said from the start this must be a ransomware attack. This was what happened to the local community college when their systems went down for several weeks a few months ago. It shut the whole school down because no one could do anything.”
North Carolina A&T State University was hit by a ransomware attack in March. Colleagues there tell me they’re still not fully functional a couple of months later.
https://therecord.media/north-carolina-at-hit-with-ransomware-after-alphv-attack/
“3xx” — would say LOL except for the reality it implies.
Ten years? They got complacent.
@WG – Anybody who’s spent enough time in IT has war stories about “good” backups.
@Gin & Tonic
I went to the site you linked and it looks like you have to have a login? Do you have a link for where you found that quote? I’m sure John would love to give them an earful.
@JanieM
re: 3xx
Yeah, i said last week that if they get to 34x, it’s definitely extinction-level.
@Gin & Tonic
Our site hosts actually test the backups, so there’s that.
*when they can get to them.
@Gin & Tonic: Yeah, I only just noticed the Stonecourt Capital thing.
Also, you are depressingly on the mark about backup war stories.
@wg oh very good to hear that the devs have at least verified the backups!
@Major Major Major Major
Yeah, I confess to quizzing them about backups multiple times, which I’m sure they appreciated. 🙂
Two off-site backups in different geographical locations sounded pretty good to me, as did their regular testing. I did not ask about air gapped backups, but of course that will be in our future.
To make today worse, i am am guess #5 on Wordle with no clue as to what the word is, and getting it wrong will completely screw up my streak.
@WaterGirl
Might I suggest … punch cards.
@Baud
No, you may not!!!
On the plus side, I managed to figure out Wordle in the last guess. I’m sure you were all holding your breath for me. //
Are we updating the graphic daily? 365…355…350…
We…FYWP out of habit…and FY320DataSecurity…
Well, their LinkedIn site job postings include a system administrator. Maybe things have been falling apart and passed the point of no return. My office had a catastrophic interruption of service with two servers at a single location, but three servers at different locations — that’s pretty well nigh insane.
I have no idea, but it seems like an extinction level event. I do know that there are some incursions where federal authorities get involved and don’t let you release information, where perhaps a federal contractor was maintaining classified information. I just have a hard time seeing that they would be using a commercial cloud provider like this one using the same resources as your garden variety blog. But I don’t really know anything.
@WaterGirl
May 23, 2022 at 4:28 PM
On the plus side, I managed to figure out Wordle in the last guess. I’m sure you were all holding your breath for me. //
________________
👍
@Baud
No better to use punch tape or mag tape. Cole would forget to set the number command on the key punch machine to number the cards and would trip over the mustard jar and throw all 12 boxes of the punched posts off the porch scattering from hell to breakfast. Hilarity would then ensue.
@Primer Gray (formerly Yet Another Jeff)
Yes. Graphic and all references updated daily.
It’s a republican attack – knowing how much money this site fundraise – the people at datacenter365 realized that they must stop it!
learn something tangible: 5/30/2022
site back up: 6/7/2022
@WaterGirl
I lost my streak at 88. It was surprisingly irritating!
@2liberal
Since you’re the only one to supply dates so far… unless that changes, I think you win no matter what. 🙂
@Benw
88? Holy crap, I’m not even going to say what my streak is.
Given the comments about firewall / access, I will guess one of the recent FT / BigIP vulnerabilities from past month, resulting in massive penetration.
There are legal discovery elements present.
And for timing: site up by Friday. But will go down for three days, back up the following week
@MobiusKlein
I have been supposing it’s likely related to the warning from the Feds on 5/11 about increased risks to data centers.
When you say “legal discovery elements present”, can you say more about what you mean by that?
I’m thinking that whoever suggested the name “365”data centers is probably regretting it.
We need a days of BJ withdrawal counter :
https://datayze.com/days-since?datesummary=May+14th%2C+2022&day%5B%5D=sun&day%5B%5D=mon&day%5B%5D=tue&day%5B%5D=wed&day%5B%5D=thr&day%5B%5D=fri&day%5B%5D=sat&date=1652511600
I assume they had some compromise that affected a law enforcement or sensitive system, and they are required to do full forensic research before opening up again. Which would explain the info blackout.
This kind of threat
https://support.f5.com/csp/article/K23605346
@Jay S in WA
I love that, but it should be since Friday because this is day 10 of the outage.
I guess there’s no way to get that live or in a form I can add somewhere on the site?
edit: I fixed the start date myself.
Here’s my wild, and wildly paranoid, idea: 365DataCenters isn’t making enough money. In fact, it’s failing completely. So they run a ransomware attack **on themselves**, hoping to squeeze some money out of their frantic clients, or the US government.
Or, Wild theory #2: I wouldn’t believe a single word they said, particularly about how all the data are still intact somewhere and it’s just a matter of re-establishing contact. I think something really awful happened that destroyed their entire system, everything is just-plain-gone, and they’re dancing as fast as they can to be out of the country before everyone who ever hosted anything on their service files lawsuits that would bankrupt them, their families, and their immediate circle of friends.
My best guess for when the mothership will be back up: June 30, on a different hosting service completely.
@MobiusKlein wrote:
“I assume they had some compromise that affected a law enforcement or sensitive system, and they are required to do full forensic research before opening up again. Which would explain the info blackout.”
Hmm. It was so annoying that they kept saying the were “investigating” and saying nothing about recovery. That would certainly explain it.
But it still seems odd/impossible that the only talk of this on the internet is a 14-comment Reddit site.
@CaseyL
Interesting speculation!
re: “on a different hosting service completely”, I can say that we will stick with our current site hosts, as long as THEY don’t stick with 365 as the cloud provider.
Quilt’s adorable!
Y’all are too effective at fund raising, we’re a target.
John and I just received this note from our site host:
START NOTE
“I had a conversation today with two senior members of the 365 team.
They have advised me that there is quicker movement and forward progress and are hopeful to provide us with a timeline for the restoration of your website(s) this week.
Based on 365’s discussions with their cyber forensic partners and their current estimates, they feel this timeline is appropriate to bring the cloud data center back into service.
Again, they confirmed their assessment that customer data is secure and untouched.
365 committed to providing a post-mortem detailed root cause analysis (after all their customers have been restored) that explains the situation and why it is taking so long to restore the environment.
I will share that report with you when I receive it. I will also provide additional updates when received.”
END NOTE
I’m not sure how to read this phrase in paragraph #2.
“…are hopeful to provide us with a timeline for the restoration of your website(s) this week.”
How do you guys read that?
Does that sound to you like we will be informed this week of the timeline for restoration?
OR
that websites could actually be restored this week?
This seems eerily similar to an outage of a major attendance/grading platform in NYC schools from back in January. They were similarly tight-lipped and evasive about what was going on, and it turned out to have been because some hacker made off with a bunch of student and family data (they’re still denying it to this day). So maybe “355” had a similarly egregious loss and is scared to admit it because that’s the end of them if they do?
Source for the NYC outage: https://www.nydailynews.com/new-york/education/ny-nyc-online-grades-down-possible-security-incident-20220112-leqn2ovkdbapvjzacfijaah5bu-story.html
WaterGirl: Apologies: I can’t keep track of who controls what piece of Balloon Juice. Have you talked to the hosting service? What would be involved in moving to a different cloud provider – and how thoroughly can you vet whoever the new cloud provider will be?
@WG – Always, always, always read any statement from a vendor in the worst possible light. They “hope” to provide you with a timeline this week. So maybe you’ll get the timeline next week or the week after.
You might be back in business by the end of June.
@WaterGirl:
I think they’re being deliberately cagey. They hope to have things up this week, but if they can only provide a timeline they’ll claim that’s all they promised.
But by all means, let’s get rid of real life and do every fucking thing in cyberspace.
Yeah, I used to *swear* that they trained the cashiers at K-Mart to be slow, because no one could be that slow unless they tried.
Similarly, re: 365 and their vague word salad… it’s hard to believe they could be that vague and unclear in critical communications… unless they run it through some special vague-i-fier routine that takes plain english and makes it all obtuse.
@different-church-lady
I have no idea what that means.
If not ransomware, then extinction-level incompetence and stupidity. No data center would be offline this long, with this bad a communication strategy, if it was even halfway competently run. I’m going to guess mid-June for being back online and also guess that, unless the media picks this up, we won’t ever get the full story. Or the truth.
This seems to by my day for bad news. I called the IRS to inquire about the refund that I’m supposed to get for the return I submitted by mail on March 4th. Alas, they are so far behind on their paper-submitted returns that they haven’t even scanned this into the system yet, much less processed it and issued the refund. They are now saying that it may take as long as 16 weeks from today for the refund to be issued.
I tried to file electronically this year but their system wouldn’t let me, probably because they also screwed up my 2020 tax return, although that one, at least, didn’t involve a refund.
@top
“Lex Luther ”
Luthor.
@PaulB
That is shitty news from the IRS. That’s 6 months from when you submitted your return. I’m sorry.
@WaterGirl: “How do you guys read that?”
I’m taking the pessimistic view: that they’ll try to have a timeline for you this week, not that they will have the site back up this week. If they were able to get everything running this week, and they knew it, they would simply tell you. The fact that they aren’t telling you says to me that they cannot.
And the fact that they still cannot commit to a timeline yet says to me that the incident is still not completely understood or over, regardless of the more hopeful tone of that message. Seriously, “quicker movement and forward progress?” That’s just meaningless bullshit.
@NotMax
Fixed.
@PaulB wrote:
“That’s just meaningless bullshit.”
I agree. Everything that comes from 355 is Sarah Palin word salad.
And everything our site hosts share with us comes from 355, because it has to be approved by their legal team.
So this isn’t bullshit from our guy, he is just the pass-through for the word salad.
Thanks, @WaterGirl. Fortunately, I don’t have any plans that depend on that refund, unlike some of my neighbors, so I can ride this out without too much pain.
Too bad I can’t charge the IRS a penalty and interest, preferably taken out of the salaries of the Congressional Republicans who have been starving the agency of resources.
@PaulB
If only.
Maybe, someone should just mention that this site is full of lawyers.
Still, in spite of the word salad, every day that they continue to say that all the data is there, untouched, is a good day.
We will not be back up this week or next.
You MAY get an estimate by the end of the week of when that’ll be.
The estimate will be highly optimistic.
Yeah, they hope to provide a timeline. It might be a short timeline, but all they’re offering right now is a plan to give you that timeline.
@🐾BillinGlendaleCA
25 lawyers all write letters in legalese to “355” on behalf of Balloon Juice, demanding more information…
355 days of data security, 355 days in service. Servers go down, buck’s passed around…354 days of data security….
Agreed. Emollient weasel words. They are trying to buy time for some reason.
I rather picked up a whiff of escape clause in committing “to providing a post-mortem detailed root cause analysis (after all their customers have been restored) that explains the situation and why it is taking so long to restore the environment.”
Keep insisting there are unspecified customers yet to be restored and that ball keeps merrily retreating into the distance as well..
@WG:
🟩🟩🟩🟩🟩 ‼️
@Primer Gray (formerly Yet Another Jeff)
That’s very catchy! 🙂
@SiubhanDuinne
I don’t speak fluent Wordle shorthand. Can you tell me what that means?
Is that a happy “you figured it out!”?
@PaulB:
Wait…you actually got through? After how long of a hold time? I’m trying to gauge your luck. You might consider getting a lottery ticket.
@WaterGirl: It’s a fucking perfect storm. Not only are the service centres just now getting their full staff back there were also a record number of paper returns filed. Right now there’s a backlog of almost 10 million paper returns yet to be processed. Add in the inability to hire replacement staff at any sort of pace and it’s ground central.
@scav
re: “why it is taking so long to restore the environment.”
Because no one ever wants to know why it’s taking so fucking long WHILE they are having to wait a ridiculous amount of time for something.
@WaterGirl
I’m not sure there is such a thing as airgapped backups. We only have tape backups if you want long term storage of data that will probably never be used again. Only airgapped systems are for the government high security servers. You can’t even pay to get in there.
Your host may be nice, but they need a new datacenter provider after this. The choices are big evil with solid security or small and probably more evil while still having the chance for this to happen again.
@ WG: She got the answer in one step! Very impressive, took me four steps.
@scav
Those RCAs are standard practice. They could be dicks about it and slow walk it, but that would probably only go on month or 2 hoping people forget and don’t care about it once they’re up. But by then most people that are going to leave will have.
For the Host provider using them its going to be a problem migrating out if they want to leave. Most customers aren’t going to have the patience, and they’ll be losing customers as well.
@watergirl I picked the 14th because the last post in my rss feed reader had tweets dated from then but they might have been GMT. Unfortunately I can’t see the post timestamp in the feed.
I wish I knew *how they know* that the data is allegedly intact. It’s not clear to me that they can know that.
I mean, what was the failure mode? Assuming that this was some kind of server-side ransomware attack (i.e. not through corrupted client data, but by infiltration of their datacenters software infrastructure), somehow some attacking process is intended to screw with the enormous filesystems on those giant RAID arrays. Obviously it can’t directly encrypt the data, so presumably it screws with some database index, RAID filesystem metadata, or some other bit of critical data access infrastructure. Then it sends a random note.
If that’s the case, the data could be perfectly “safe”, in the technical sense of existing, unencrypted, somewhere in a vast pile of storage. But we would nonetheless be screwed, because the cost of *finding* the data and restoring everything would bankrupt the company. Unless they pay the ransom, that is.
In any event, in this scenario the statement “your data is safe” would amount to saying “we have no reason to think that any corruption occurred to it”, NOT “we checked it and it’s OK”.
I’ve got my Firefox phone browser set to clear my cache any time I fully close it. Every couple if weeks, I hit the quit button instead of just soft closing the app and the Wordle data is gone. Current streak 2 for 2. Solved today’s in 2.
Irin Carmon
On the cover of the latest
@nymag
: This Magazine Will Help You Get An Abortion. A wildly ambitious, incredibly useful, regularly updated access map and much more practical reporting from
@thecut with an introduction from me
https://www.thecut.com/article/future-abortion-access-map.html#_ga=2.244201733.1094371790.1653309671-1292490262.1649268386
I’m kind of pathetically grateful that at least one magazine thought this was important enough to cover, not the politics, not the ridiculous SCOTUS cloak and dagger, but the real issue.
https://web.archive.org/web/*/https://balloon-juice.com/
@Urza
Our site hosts are just a pissed as we are, so with the abysmal communication from 365, I can’t imagine they will have much interest in sticking with 355.
But that’s a conversation for after everything is up. We will have that conversation, though, for sure.
re: air gapped… In this context, I took M4 to mean “not part of the same system”… “in a place that is totally separate from the operating environment”.
@debbie
I feel like getting it in one is sheer luck in picking your starting word. Very fun, but more luck than skill.
Getting it in two requires more skill, I think .
@Jay S in WA
We were up in the morning on Saturday but we were down by about noon. So I consider Saturday to be the first day of outage, making Friday 5/10 the last day of service.
@WG: // Is that a happy “you figured it out!”? //
Yes, that’s what I had in mind!
@Carlo G.
We don’t know if it’s ransomeware or if someone just screwed with the virtual routing at the data center (early on they told us there were issues with virtual routing) or… or… or.
They have outright stated that they have verified that the “security incident” didn’t reach any of the systems themselves.
They outright stated that they have checked backups and verified that they are fine.
Given how cagey they have been, I think the actual clear statements they have made, however few there have been, are actually truth.
@SubaruDiane
Have you seen Word Master? It’s essentially a Wordle you can play as often as you want. I binged a couple weeks back, spent so long on it I have strategies.
@SiubhanDuinne
I was ridiculously pleased to have gotten the word in the end.
@PaulB, thank you for the update on late tax refunds. We’ve also been waiting for ours since March.
And we also had problems with our return last year, but I do have a somewhat good news story about it: we made an error last year on our return, so they sent us a letter telling us we owed $1800 more than we had paid. A quick review of our documents revealed the error, so we filed an amended return, and waited for an update. After a couple of hours on hold, I spoke to two different IRS reps, who explained that we still owed the money while the amended return was pending and in process, but we could get an extension of the time to pay the penalty.
We got an extension until November, but by October, the system still showed nothing about our amended return. Not wanting to risk getting a tax lien, we just bit the bullet and paid the money. But early in April, lo and behold we received a refund check for the penalty amount, plus letters indicating that the matter was resolved.
So I have faith that there are people working there who know what to do; there just are apparently nowhere near enough of them. Naturally, I blame TFG for gutting all our agencies.
https://web.archive.org/web/20220510033326/https://www.balloon-juice.com/
Looks to have a pretty complete run of the blog through May 10, so may worse case is a loss of 3-4 days?
@Kay
Irin Carmon is terrific here. More of this from our media, please.
@WaterGirl, et al.
Mildly curious if Wordle accepts COVID as an entry.
@NotMax
I will give that a try for you sometime when I have O and I for vowels. Will let you know.
Any competent cloud service would say “your data ARE safe…” since the word “data” is a plural noun.
@Jerzy Russian
That assumes that the tech people are writing the communications, which would never happen.
I have three start words for Wordle; “Liars, toned, and chump.” Covers the 15 most common letters in the alphabet including every vowel. Depending on how productive the first is, I may deviate and pick words including those other letters, but since I started this I almost never fail a wordle.
@jersy data’s been singular for years, sorry.
I want that quilt.
I want that quilt. My cat has made ALL those faces
@M4 I was making a funny, of course.
The original meaning of “data” is the plural of “datum”. My dictionary says that “data” can either be singular or plural. The scientific journals in which I publish generally use the plural, and their copy editors will enforce this style.
@Major Major Major Major
People have been fighting about that for years, but i have no dog in that fight.
“We’ve managed to put our hands on this datum of yours, and are digging industriously through the break room couch cushions and the NOC’s junk drawer to locate more.”
@NotMax, @WaterGirl — Wordle does not accept covid.
@WaterGirl
Yours truly is among the last still using the plurals stadia and aquaria.
😉
@Yutsano : Regarding getting through to an actual IRS agent, the process took 8 tries. The first two resulted in phone system dead ends that told me nothing and offered no option to speak to a real person.
For the next several tries, I looked up the method online for phone system prompts that would get me to a real person. The first five attempts using this method led me to a “We’re too busy right now. Goodbye.” message.
Finally, on the 8th try overall, I was placed in the queue, where I waited for about 50 minutes to talk to someone. She was, thankfully, reasonable, apologetic, and helpful. Well, as helpful as she could be, given the circumstances. She was able to tell me that the issue with last year’s return had been resolved, so that was good.
I offered her my sympathy for the calls she was handling, and will continue to handle, until all of this year’s returns are processed. No fun for her or for us.
@Layer8Problem
“”We’ve managed to put our hands on this datum of yours, and are digging industriously through the break room couch cushions and the NOC’s junk drawer to locate more.””
Worthy of XKCD.
I can only aspire to a tenth of the wit of one of his hover-over alt-texts. How the hell does he do that three times a week?
We got a letter from the IRS stating we (well, I) screwed up our child tax credit and that we are due more than we thought.
They gave me a number I could call and dispute that, but I will pass.
Thanks, TurboTax
And, that letter from the IRS was dated today. I BET THEY WERE USING OBAMA’S TIME MACHINE.
“”I can only aspire to a tenth of the wit of one of his hover-over alt-texts””
I never even thought to check that. Thanks for teaching me something.
So either their route tables got hi-jacked or they got so far in arrears on their ISP billing that they are radioactive.
They are letting everyone assume the former – it sounds a little bit better from a p.r. perspective, but acting to the latter.
I suspect they’re lining up buyers for the customer list and physical assets and sinking the hulk under a chapter 11.
@WaterGirl: That means they’re going to tell you this week when you’ll have access. It doesn’t mean that plan won’t finish this week. but they’re not committing to restoring access this week.
I work on this stuff for a huge company. It might not be ransomware, but at minimum they got owned by a hacker in a way that makes ALL of their security and token suspect. This isn’t quite the end of the world for a data center but i imagine it would cause enormous amounts of damage to their business.
We’re liable for damages if a customer is down for more than 12 hours. This is way in excess of that.
@Kropacetic: You’re welcome. Don’t recheck all 2633 of them at once. 🙂
I have a full site crawl (maybe more than one) back to 2003, of the index.htmls for articles on the front page, including comments.
FWIW. (I mostly use a rendered (lynx –dump) copy but have the htmls.)
Ping me if needed (either my registered main or preferably my nym (space to dot) at gmail.com).
@Bill Arnold: The difficulty is that to restore a WordPress blog, what is required is both the WordPress site file containing the site formatting (minor, annoying, can be redesigned) and the WordPress database, containing all the content. So it’s not enough to have the content: it ha to somehow be poured into a WordPress MySQL database, with the correct table structure. This is a highly nontrivial programming task requiring some IT ninja-level skills, and probably a shitload of trial-and-error-and-error-and-error-and-error.
Which is to say, it is a very, very last resort, and if we have to go there, we Jackals will be pooling all the tech foo we have, and not a few resources.