Also, Water is Wet and the Sky is Blue

This is TERRIBLE news. They have given up on retrieving the data. They are offering to rebuild your environment if you have a backup to provide them. That’s how I read it. They are going out of business for sure so there will be neither the time or money to retrieve the data even if that were possible.

That letter, my god, is Bob DeSantis James Cornman some AI? Who talks to angry customers like that.

> We will continue to be transparent

“Continue”?

EVT, I read that last name as “Conman” several times before I realized that it was CoRnman”.

That’s some catch, that catch 22:

We are now able to confirm that the May 14th security incident was a ransomware attack. We are also able to confirm that neither 365 Data Centers nor our customers were the target of this attack. The intended target was a third party whose data is stored in a dedicated environment on our cloud platform. Unfortunately, for our valued customers and 365 Data Centers, the cyber-attacker broadened the ransomware attack.

About what we figured, but blargh, not the greatest news. Still, thanks for the update, John. I’m on board for whatever is next for this community.

Also, if $$ is needed to… Whatever, feel free to rattle the cup. Pet treats, ice cream, bird feeders, coffee… I appreciate *your* transparency.

@Watergirl – I’m retired, but in my work life I knew a lot about internet routing, and this makes no sense at all.

@Gin & Tonic:

What are the consequences 365 could face in your professional opinion?

@Goku – Bankruptcy.

@Barbara – Bingo!

@Watergirl – A virtualized route reflector means you’re too cheap to have an actual (hardware) router doing the job.

@Watergirl – It’s real routing, not virtual, but it’s late and BGP is complicated so I’ll bow out.

@Edmund Dante’s: Srsly. No *secured* backups? What kind of chimps run this company? The letter is an impressive example of bullshit, tho.

@Gin & Tonic
While virtualized reflectors are there to save money, large datacenters don’t actually require that many physical routers. Generally 1, maybe 2 per rack is more than sufficient. Then another layer for every 10-20 racks. Scrambling the DNS tables wouldn’t take much, if you’re paying attention Amazon and Microsoft seem to be able to make that stupid mistake about every 6 months without any outside help. They fix it in hours though.
The legalese of the data maybe not being accessible is hopefully just about them covering themselves instead of making promises. But they could easily have lost some drives or other hardware in the last 2 weeks and not been able to tell with everything scrambled, that would also prevent data replication to prevent losing the last copy. Usually you find out all the server problems that were building up over time when you start booting everything back up, even though it may have seemed to be running fine before it went down.
This kind of mess is what I call a long Tuesday. The difference between the big players and smaller cloud providers is in how much it actually affects the customers. Big provider maybe you notice a blip as the network starts pointing somewhere else, maybe it even goes down for a few hours. Actually allowing more than a few servers to stay offline with customer data for weeks is insane. Have had drive heads crash and dig furrows through the platter and we still got that data back even if it takes awhile. Anything else is unacceptable. Whole datacenters offline should be shooting up news flares everywhere if 365 was actually a company anyone gave a damn about. And our CEO would be on that call every waking hour making sure it got resolved as fast as possible for something this large.

I dropped a link to a tarball (7GB) of a crawl of the top level posts that goes back to 2003 (through may 13 midday of 2022) in a thread last night. Not everything, e.g. no images, but at least the text and comments, as html. If there are any gaps, I might have a few other crawls – would need poke around. (Am a packrat.)
I’m hoping the site can be restored in full, though.
Here’s a copy-paste:
—–
Here’s a google drive link to a tarball (tgz) of the crawl of top level posts/comments (back to 2003) that I mentioned yesterday.
https://drive.google.com/file/d/10NwLTn1krg2ID0sd76xerP3JUFDji0XK/view?usp=sharing
I’ve downloaded it and tested (unpacked it) and verified that it’s the one I made.
Just in case it is needed. It could form an archive html site at least, though Carlo said that it is not nearly sufficient to rebuild the site.

@WaterGirl

Wait, there’s actually a class action lawsuit against 365? I thought that was just Cole’s snark

OK. Keep in mind that there is a big difference between restoring the live service, which appears to be what 365 is mostly interested in doing, because that keeps their customers inside the fence, and…

THE FUCKING BACKUPS, WHICH HAVE NOTHING TO DO WITH ANY OF THIS NOISE, AND WHICH THEY *STILL* COULD TURN OVER IN A FEW HOURS, AFTER WHICH BJ COULD BE BACK, GOOD AS NEW, WITHIN A FEW MORE HOURS.

Jesus, this is more misdirection. Please. Can we keep our eyes on the backups, and not on these idiots attempting to restore their incompetent service? Every time they issue any fucking statement, please, please read it and ask yourself “does this have *any* relevance to 3xx turning over the backed-up BJ data?” If it does not, then the only way forward is still the lawsuit.

Stiff upper chin is an exquisite mixed metaphor. Given that I’ve packed on some pounds recently, my upper chin would be th only stiff one. The rest are a bit flobby.

@Bill Arnold: That probably won’t fit on my Commodore 64 webserver.

They certainly moved the goalposts from

“we’ll be right back after this brief interruption” to

“nobody stole your information, but some of you have lost your apes”

I’m not proud of it, but one intentional tactic I use when working with clients in a potential data loss situation, if there’s any reasonable hope of recovery, is to tell them that it’s under control until the moment at which I conclude there’s nothing further I can do. My rationale is that the client is already freaking out, and I can serve them best by offering them confidence and safety for the duration of my efforts.

And, in the unlikely (but occasional) event that I really can’t save the day, the impact of the bad news would hardly have been lessened than if I’d been totally transparent with them from the start about my lack of certainty.

IOW: I’m not really sure that 353 was being totally straight when they said the data is safe, and I never really was after the third day. They were hopeful, maybe even reasonably so. But the above communication seems closer to saying that they’re not as hopeful anymore, and it’s time to start bracing for impact.

The big difference between me and them is I don’t promise or provide uptime, backup, security, or hosting, and that’s on purpose. I don’t want that kind of responsibility or liability. I will make best efforts to enroll people with best providers and systems, but lord knows I don’t want to have lost data to account for. If you’re going to run a business whose whole point is to store data that isn’t your own, you’d sure as shit better be bulletproof against data loss.

I am no longer optimistic, if I ever was, that we’ll ever get anything back out of 353 Data Centers — not the old site baxk, not the archived data, not a penny of financial compensation for John Cole. 353 Data Centers simply isn’t acting like it can do these things.

This is what I was afraid of. The backups are irretrievably corrupted, and 3XX does not have the resources to unscramble and/or decontaminate them.

The only question is whether 3XX can find a clean backup anywhere. Maybe yes, maybe no.

Well, isn’t that a charming pile of hooey from 353.

Hot! Damn! Glad I’m not longer in that goddamned game.

@NotMax Yes, assuming they have uncompromised backups. But given that after two weeks they have not done that, and given the wording of that email, they have no uncompromised backups. It sounds like they stored their backups in the same data centers that were attacked.

There go two miscreants
Yea, it would be awful if they lost my email and phone number! /s

I am not sure I believe what they say because I really don’t trust them any more.
If this is the truth, my first thought was why wasn’t 356 paying the ransome even if they weren’t the main target, and then I realized maybe the ransome demand wasn’t just money, but something else such as information and I realized again how bad ransome are attacks are.
I have also been surprised there haven’t been more Russian cyber attacks on everything since the Ukraine war heated up. I was kind of expecting more in the news.

So: the guy being held up on the blog’s behalf is named Dave, and 3xx is addressing him as if it were HAL 9000 suggesting a stress pill.

The next scene should be cherce. Hope Dave supplies video.

The supposed 353 insider on the Reddit thread said that he thought that it would take weeks to months to resolve the issues. That tells me that the problem is solvable in a technical sense. It seems to me that the issue is financial and whether the MotU that own 353 figure that it’s cheaper to declare bankruptcy and walk away or to actually do their jobs and spend the money to recover the information on their boxes. IOW, will 353 be in business in the weeks/months that it would take?

I’m up for helping to pay for B-J 22.0 – Better, Stronger, Faster. Waiting for 353 seems to be delaying the inevitable, it seems to me. They have only been spouting self-protecting legalese and clearly have no concern for the people with stuff on their boxes. There’s no objective reason to think that will change.

Follow the money.

Good luck to us all, and hang in there.

Cheers,
Scott.

@ debbie May 26, 2022 at 8:17 AM

I don’t really understand any of this either but I am grateful that so many knowledgeable jackals are posting their explanations.

I posted this link and snippet that I found on a google search a few days ago. (The link doesn’t work anymore. Duh). The marketing team of 353 data centers pushed their company as a leader in dealing with ransomware attacks. Maybe they should have checked with the company before offering that service. 🙄

Understanding Ransomware Attacks: How Cybercriminals Infiltrate Your Mission-Critical Data?
Oct 5, 2021 — All this information may seem daunting, but 365 Data Centers is here to help and protect your data. With over a decade of experience in the . . .

Watergirl:

“The way I make sense of *that* is that they can confirm that the hackers didn’t get into any of the not-the-double-secret-third-party areas, but the hackers so messed up the routing that “353” Data Centers can’t retrieve it.”

Maybe, but industry experience and cynicism make me suspect that the translation from business-speak/legalese is more like this:

365 DC: “… evaluation to date by our systems team and cybersecurity experts has revealed that, aside from the targeted third party, no data was taken from the 365 Data Centers cloud environment …”

Translation: “We think they encrypted everything as quickly as they could access it, meaning they probably didn’t scan for credit card numbers and personal information. Yet.”

365 DC: “… 365 Data Centers believes that at this point in time the prudent path forward is to rebuild the affected cloud platform.”

Translation: “You’ll need to sue us to get your money and data back.”

365 DC: “This will be conducted along with an all-out effort to retrieve all data within the existing cloud environment that can still be accessed.”

Translation: “We have some backups from 2019. If any of your files are among them, we’ll give them back to you in exchange for a settlement where you indemnify us against any other damages. We don’t have anything more recent than that, because we stopped doing any due diligence or maintenance in 2020, figuring we could blame any problems on Covid.”

365 DC: “If your preference is to work with us to restore your service on a new 365 platform, please inform Steve Oakie, 365 Data Centers’ Chief Revenue Officer.”

Translation: “Sure, we’ll be happy to bilk you for more money if you’re dumb enough to keep doing business with us.”

365 DC: “We are saddened by the impact this incident has caused on our many years of collaborative hard work with you to build your cloud services. Our entire organization is sorry for the significant inconvenience that this has brought to you and your business.”

Translation: “You’re on your own. We’re too busy working out how to monetize our remaining assets and distribute the proceeds among management, in the form of golden parachutes, before we declare bankruptcy and dissolve the company. But, really, we do feel bad about it.”

@JR in WV: At the data scales in question here, it is literally not possible for an attacking process to encrypt the data. There is simply too much of it. The most practical approach is to lock databases containing metadata, mess with the layers of management software far, far above the RAID array management that make data administration practicable, on the theory that even though the data is safe and sound in those RAID arrays, and recoverable in principle, as a practical matter doing so without paying the ransom would bankrupt the company.

After re-reading that scurrilous and laden with falsehoods letter, I want to quote just a tiny bit to illustrate the inherent contradictions these 365 guys published in their f’ing letter:

“We are also able to confirm that neither 365 Data Centers nor our customers were the target of this attack. The intended target was a third party whose data is stored in a dedicated environment on our cloud platform.”

If “third party” had data on 365 DataCenters cloud platform, they were in fact customers of 365, unless there was some other contractural relationship between “third party” and the 365 people. Like the guys’ coke dealer’s customer database or something like that.

Totally not professional to frame up one customer as anonymous third party, also too!! Now I am wondering really hard what the ransom demand to decrypt “our” data was… and how did these ignorant arrogant wankers respond to the hackers demanding that ransom?

Because pissing them off can convert the “ransomware” attack into a plain old “we destroyed your data because you pissed us off so bad” attack.

When did the feds (and which feds, it’s a large group!) get involved?

You don’t make kidnappers holding your daughter angry because they can cut parts off her and mail them to you.

@J R in WV — If I had to guess, it would be something like their customer is a contractor to some other entity that was the target, and the contractor was storing that entity’s data as part of its services to the entity. I would also venture that the targeted entity is either a government agency or under contract with one. I would further speculate that is a big reason for the silence and apparent secrecy. Not every customer carries a threat of law enforcement.

And this is just a random fact of interest, entities are now being sued under the federal False Claims Act for falsely certifying that they meet certain types of cybersecurity standards when they clearly don’t. These kinds of companies are not used to being regulated, even minimally. They aren’t typically prepared for this kind of scrutiny of their practices.

@debbie: I’ll second that remark from @The Moar You Know, and add a datum. Back in the early 2000s, it was still possible to administer a networked Linux/Unix computer and keep a process watching the system logs for attacks. I had such a process, which would send me an email with the IP address of the attacking host. This would happen maybe once a week or so, at first. Annoyed, I would go to the trouble of looking up a sysadmin at the University or corporation housing that host, so that they could walk around and give the offender a talking-to.

Within a year or two, maybe 2003 (memory fails me) the attack rate (probe rate, really) was already so high that this was an absurd approach. Automatic log-scan-and-firewall-jail solutions were becoming mandatory, because logs were filling up with *mostly* entries showing probes — essentially people trying doorknobs to see if they could find an open door.

That was 20 years ago. Nowadays, an appreciable fraction of all traffic on the net (well by connection, if not by data volume, given the size of video streams) is malware. And system logs of any host connected to the net and reachable outside a firewall show connection scans across ports from all over the world, all day long, every few seconds, or minutes at the longest. Much of it is botnet herders looking to grow their flocks, presumably. But some is target-of-opportunity—you never know what you might score on a maladministered, high-value system…

And, since I’m having an Old Fart’s IT Cockups In Days Of Yore moment, here’s a story about what may have been the earliest, certainly the strangest, oddly inadvertent DDoS attack in internet history (WaterGirl has heard this one already, in the dark early days of the 3xx outage).

In 1994 I was a postdoc at NASA’s Goddard Space Flight Center, working in High-Energy Astrophysics. The Internet was mostly text back then, but this new-fangled http thing that Sir Tim had bodged together at CERN had gotten kind of popular, and Netscape was Cool and The Future.

NASA, never slow in the PR game, was trying to figure out how to use the web get as much visibility as it could. Astronomy Picture of the Day was born around that time, and Hubble pictures were especially popular content. However, the idea of decentralized content servers had not yet been introduced, and all the traffic from the population of future space-boosters that NASA management was trying to grow was being driven to web servers at Goddard, through a single largish (for the time) but finite-capacity pipe intended for research communication and data exchange.

You probably see where this is going. Comet Shoemaker-Levy 9 made world news in 1994 when, after a previous orbital pass at Jupiter had broken it into numerous fragments, calculations showed that those fragments were all heading for a collision with Jupiter itself, some time in July. NASA set up several space-based assets, including Hubble and the Galileo Jovian mission, to observe the events, and basically promised a livestream, possibly the first of its kind, certainly of its—unintended—scale.

Over the course of a week that July, as each fragment in succession flung itself into Jupiter, releasing a total amount of energy estimated at roughly 600 times that of the Earth’s entire nuclear arsenal, tens of millions of people from all over the world attempted to get a glimpse of what was going on through that data pipe. Which to us scientists now performed much less usefully than the 28kBps modems that we had at home (DSL broadband was a distant dream then). I felt that we’d been DDoSed by Dan Goldin, the publicity-hound NASA Administrator, who had the tech savvy of a goldfish with a learning disability. We just had to wait for Sir Isaac Newton to sort things out—eventually the last fragment dropped, the public got bored, and we got our network back.

Looking at this sentence:

“The intended target was a third party whose data is stored in a dedicated environment on our cloud platform.”

According to them the target was someone they were hosting and it was a ‘dedicated environment’.

Except either it wasn’t actually dedicated OR they had sloppy security which allowed the ransomware to cross out of the ‘dedicated environment’ OR they had a person who had sloppy security that accessed 1 infected environment and cross infected another.