Lawsuit Filed & Site Update

WG, amazing work. Wow. !!

Happy news indeed. Many thanks.

I don’t say this lightly, but god bless you, WG.

These things will cost money. Let us know when. Also get a donate button or something up. Really.

Thank you and the front pagers for all of your hard work. Many thanks to John for keeping this place open. I visit other places, but this place remains far and above my favorite. There’s a community here. So whether we stay here or move back to the old place, I’ll be tagging along!

O frabjous day! As the kids say. This is definitely welcome news.

Rest easy, WG and crew. Thanks so much.

Cheers,
Scott.

You fit in 3 hours of gardening too!!?!??

Where would we be without your hard work? Many many thanks!

@Yutsano

And may I just add, “Calloo, callay!!”?

thank you everyone for the hard work, I have missed the old site (even though this temp one is awesome).

My weekend work was with our network admin and installing new switches at our office and re-IPing the storage array and all the servers. Scary times indeed.

:-*

WaterGirl is the best fixer in the blogosphere.

Awesome news!! I’ve missed BJ, especially OTR with Albatrossity and Bill. Looking forward to seeing the site!

“Jackie May 28, 2022 at 8:08 PM
Any monies won should go to the BJ account.”

Speak for yourself! I expect compensation for pain & suffering and reimbursement for self medicating during this out(r)age!

Edited to delete the /jk tag in case they use that against us.

Thanks for the good news and all the work.

WG, you are an absolute treasure!

Thank you WaterGirl.

Wonderful news! Thank you, WaterGirl — you are a Treasure!

Youse are ridiculously awesome and WaterGirl deserves a medal.

WaterGirl, thank you, thank you, thank you. Amazing job!

@Bill Arnold: It would be worthwhile thinking through a design whereby a limited stack of current BJ posts and related comments — one year deep, say — gets continuously and automatically refreshed, with older stuff going into a static, searchable archive that does not burden the WordPress database (and gets deleted from the database) as new stuff comes in.

Then it becomes trivial to scatter archive copies across diversified providers for robustness against disasters, and the current BJ never turns into a waddling hog constantly dragging along vast indexes of data that are never consulted.

This all seems quite scriptable to me. If you think it practicable, perhaps your prototype could be designed with a view to extension to later posts and data.

We get to go home on Tuesday!!! I’m sooo happy! This has been a great temporary home, (Thanks, WaterGirl!) but there’s no place like HOME!🥰

Echoing all the thanks! You and y’all are tremendous.

Thank you to all who are working on getting the site back. WaterGirl, you are once again heroic.

Thanks to the BJ front pagers for all the work . BJ rises like a phoenix from the ashes.

@WaterGirl: I didn’t know that, being relatively new around here. Are there specific reasons that might be addressed by a reasonable design? After all, using the live WP database as a search engine for old material is not particularly efficient, and has very obvious performance costs for current discussion. It seems likely that a separate lightweight searchable archive could be presented transparently in BJ as if it were almost integrated with WP through a bit of web design magic, and the benefit would be a big load off the WP database.

Dear Water Girl
Thank you. You and the others do so much for all of us here on this nearly top 10,000 blog.
Balloon Juice was very much missed while our train was crossing Eastern Montana. There’s no wifi on the Empire Builder between Chicago and Seattle, but I managed to use my phone as a hotspot most of the time.

Via a comment in Cole’s twitter thing, here’s a page with a link to the complaint:

and here’s the 72 page .pdf.

Cheers,
Scott.

update: links removed from here at moved up top because at least one of them was breaking the margins. thanks, Scott, for the links.

From page 23 of the Class Action complaint:

=== quote ===
72. In their everyday business operations, and as an integral part of their business, Biz place significant reliance on its ability to access and transact with the products and services provided by 365.

73. Biz also purchased additional “backup” services to ensure that his customers would not lose access to their webpages and Critical Infrastructure after a disaster.

74. Defendant has now informed Biz that it cannot access the backups for which Biz paid Defendant. Upon information and belief, these backups are either encrypted by the cyberattacker and/or have been deleted from Defendant’s systems.

=== unquote ===

Unsurprising, but unfortunate.

🙁

Grrr…,
Scott.

Oh. Shit.

“Defendant has now informed Biz that it cannot access the backups for which Biz paid Defendant. Upon information and belief, these backups are either encrypted by the cyberattacker and/or have been deleted from Defendant’s systems”

3xx lied about the backups. Smiled, and lied.

OK, it’s on to the ridiculous plan. We have to figure out how to shoehorn that archive back into a WP database — and how much of it is worth shoehorning.

Wow. Never checking backups for accessibility and reliability is something I espect from, say, 15-years-ago me as an individual when I had a drive crash and hadn’t backed up in 6 months and it was corrupt anyway.

Nowadays I do nightly backups using two completely separate systems in my home plus a commercial service offsite. And all I’m safeguarding is my personal stuff. Shocking that a professional outfit wouldn’t rise to at least that standard.

What they are asking for:
REQUEST FOR RELIEF
WHEREFORE, Plaintiffs, individually and on behalf of all Class Members proposed in
this Complaint, respectfully requests that the Court enter judgment in their favor and against
Defendant as follows:
a. For an Order certifying the Nationwide Class and appointing Plaintiff Biz as class
representative for the Nationwide Class;
b. For an Order certifying the Nationwide Subclass and appointing Plaintiff PaleoMom,
Plaintiff Core Wellness, and Plaintiff Speed School as Class Representatives for the
Nationwide Subclass;
c. For an Order appointing Plaintiffs’ Counsel to represent such Classes;
d. For a finding that Defendant has been negligent;
e. For a finding that Defendant breached its contracts with the Nationwide Class.
f. For a finding that the Defendant has engaged in unfair or deceptive acts or practices in
the course of trade or commerce which constitute violations of the Connecticut Unfair
Trade Practices Act;
g. An Order preliminarily and permanently enjoining the Defendant from the use of acts
or practices that violate the Connecticut Unfair Trade Practices Act, including, but not
limited to, the unlawful acts and practices pleaded in this Complaint;
Case 3:22-cv-00715 Document 1 Filed 05/27/22 Page 45 of 72
46
h. For equitable relief compelling Defendant to utilize appropriate methods and policies
with respect to ransomware protection;
i. For equitable relief requiring restitution and disgorgement of the revenues wrongfully
retained as a result of Defendant’s wrongful conduct;
j. For an award of actual damages, punitive damages, and compensatory damages, in an
amount to be determined;
k. For an award of costs of suit and attorneys’ fees, as allowable by law; and
l. Such other and further relief as this court may deem just and proper.
JURY TRIAL DEMAND
Plaintiffs demand a jury trial on all issues so triable.

So I see a request that they go forth and sin no more, a request for monetary relief, and a public trial.
I don’t see an explicit demand for data recovery to the extent possible.
IANAL

Okay I left out the opening paragraph that might be interpreted as data recovery:
“173. Under the provisions of Connecticut General Statute § 42-110g, Plaintiffs are entitled to and seek to recover actual damages, punitive damages, and such equitable relief as the Court deems proper, including attorneys’ fees. These forms of relief are in addition to and not a substitute for the claim for restitution and other equitable relief alleged in this Complaint. ”
Other paragraphs about restitution mostly talk money though.
All in all I think Biz has given up hope of data recovery.

Well, let us parse:

“Defendant has now informed Biz that it cannot access the backups for which Biz paid Defendant. Upon information and belief, these backups are either encrypted by the cyberattacker and/or have been deleted from Defendant’s systems”

365 tells Biz it cannot “access” the backups. 365 did *not* say that the backups were either encrypted or deleted. That is inferred by Biz “upon information and belief”.

But in fact it’s not that plausible, it seems to me. The attack was probably on hundreds of terabytes, maybe petabytes of data owned by several entities, not just BJ. It is impossible to encrypt that much data surreptitiously, and not that easy to delete it. I still think it’s likely that 3xx told a version of the truth: the data is fine, somewhere in a giant maze of RAID arrays, but the ransomware attack scrambled the high-level metadata that they need to figure out where it is, and methodically going through their disk farms and reconstructing that metadata is a bankruptcy proposition for them.

However, if this is in fact the case, then perhaps this would be a relief that the court could order.

Thanks @WaterGirl, John, and the whole crew for keeping the lights on!

Appreciate all the hard work you, John, and everyone else involved is doing to keep this community active.

@WaterGirl: “Encrypted or deleted” is Biz’s inference, not established fact. I feel that it is much more likely that the data is safe (as 3xx claimed) but unlocatable because the attackers screwed with their infrastructural metadata that allows them to know where everything is in their maze of filesystems (hence making 3xx’s claim misleading).

Again, the difference is significant. If this interpretation is correct, 3xx may or may not be willing to launch the recovery due to cost, and may attempt to protect itself from the recovery cost via a bankruptcy proceeding. The court could, perhaps, preempt such a course.

On such legal matters I must say I know little to nothing of what is possible, though,, so perhaps the copious legal talent hanging around could comment.

@Ivan X: Without wishing to express certitude, I believe that this is a very real possibility.

The thing is, people have a mental model of ransomware attacks — compromise a host, encrypt the data, send ransom note — that precedes the cloud era. It is still relevant to cloud hosting, but it’s what you might call a client-side attack: you compromise a customer account through a phishing email, say, then move in and use classic ransomware technique. The cloud aspect is irrelevant here, because the hosting/storage is an abstraction from the attacker’s point of view.

The 3xx attack was not a client-side attack. It was a server-side (or maybe “infrastructure-side” would be a better term) attack. The attackers seized control of 3xx’s infrastructure control layer, and did something to screw up 3xx’s ability to maintain their service. Allegedly reversibly, upon payment of ransom.

OK, so what could that something be? Deleting or encrypting data is out of the question, in my opinion, because it takes too long for petabyte-scale data, and would have consequences that are easily detectable and interruptible by IT staff. Moreover, to be an effective blackmail threat and not just vandalism, there has to be a technically credible reversal method. You can’t just bullshit professional IT looks and their cyberintrusion specialist consultants the way you could bullshit some PC owner at home.

One simple way to do that is export some critical database describing the master filesystem layout, encrypt the export, and delete the database (and any backups of the database, which you locate by tracing its backup system). Done. Data is safe, encrypted db backup in possession of 3xx, now just send ransom note: “We send decryption key of db backup on payment of…”, definite articles dropped due to translation from Russian. You could even send a transcript of the shell session in which the encryption and deletion occurred and add a hash (unforgeable file signature operation) to that session, so that 3xx could verify the hash of the encrypted db file and hence have further assurance of the technical validity of the reversal procedure.

I’m no IT code slinger, but what gets me about this whole saga is that it never should have been possible in the first place. Client virtual machines are supposed to be isolated from one another, so ransomware on one VM should not have affected anything else. That seems to imply that, contrary to 3xx’s protestations, that they were the target of the attack (either through negligence or incompetence). IOW, Boris in St. Petersburg locking up the FYWP dB for one of Biz’s clients via a ransomware attack could not, in any sensibly run outfit, take out 3 cloud datacenters.

There’s still too much of this saga, and 3xx’s comments about it, that does not make sense. Maybe it’s some variation of the Kaseya or SolarWinds attack, or doors that weren’t closed after everyone should have known about those vulnerabilities….

https://purplesec.us/kaseya-ransomware-attack-explained/

Dunno.

Lots of lessons to be learned after this saga.

Cheers,
Scott.

@Another Scott: From the fact that 3xx reported trouble with their BGP network infrastructure (the “virtualized route reflectors” stuff) as a factor in the intrusion detection, it is completely clear that this was not a classic client-side ransomware attack. It was a direct attack on 3xx’s cloud infrastructure, bypassing clients, so that VM isolation is not relevant.

3xx just had shitty security. Pretty sure you couldn’t attack AWS this way.

@Another Scott: The “target” of the attack, and the *vector* of the attack are two distinct things. As you correctly noted, it would have been literally impossible to take down the entire cloud infrastructure through a client-side attack inside a VM. That’s just the nature of VMs. It was completely clear from the earliest developments, and the information divulged to WG by David of Biz, that this was certainly an infrastructure-side attack.

Spinning it in terms of who the target was may have been an effort to deflect attention from 3xx’d gross incompetence and security malfeasance.

@Carlo Graziani thanks. Makes total sense.

But, if that’s the case, and we assume 3xx was indeed the target and not some third party (who maybe was the real target, but 3xx was the means), then what’s the holdup? There ain’t no getting it back without paying. Are they dicking around negotiating? Are what the attackers asking for so far beyond the financial pale of what 365 can even pay that they’re stuck? Are they busy trying to decide whether bankruptcy with all their clients burned but some cash left to pay bills is better than bankrupt and broke?

And, if they can’t, is there in fact any means of reconstructing RAID/SAN, file system, VM, and backup metadata from raw disk blocks, which may or may not have been locally encrypted by 3xx in the first place, at datacenter scale? Seems impossible to me, but I’m a personal computing consultant; I don’t work at that level.

@Another Scott — interesting complication indeed. Wouldn’t it be possible to convert dollars to some mutually acceptable fiat or crypto currency? They probably want to be paid in crypto anyway. Or would that still be illegal?

@Ivan X – no idea about the details – IANAL.

ProPublica [eta-not sure what’s going on here with user-links, but it looks like anchor-tags work, eventually.]:

=== quote ===
The lack of clarity puts the onus on victims to discover if their attacker falls into a sanctioned category. Determining whether groups are operating out of North Korea or Iran, for example, or on behalf of the FSB is “very, very challenging because there’s obviously efforts to conceal that on the other side,” said Ryan Fayhee, a sanctions attorney who works with victims. The government makes it seem “as if this is a traditional commercial enterprise and you can just simply screen the criminal,” he added. “That’s not how it happens.”

The federal government has long discouraged the payment of ransom and in recent years has put the professionals who work with ransomware victims on notice. In October 2020 the Treasury Department issued an advisory saying that “companies that facilitate ransomware payments to cyber actors on behalf of victims” may “risk violating OFAC regulations.” A second advisory, in 2021, seemed to acknowledge that victims sometimes make payments that violate sanctions. In those cases, victims and their representatives may receive leniency if they quickly report the incident and payment to OFAC.

Since many victims in the past have been loath to report attacks to the FBI, fearing that the intrusion would become public or the FBI would instead investigate the company itself, the Treasury Department hoped the guidance would prompt more victims to work with law enforcement. That, in turn, could lead to more indictments and more sanctions.

That part of the strategy seems to be working: More victims are reporting incidents to law enforcement, according to Waters. Following the 2021 advisory, many insurers began requesting proof that policyholders making ransomware claims report the incidents to the FBI, he said. The insurers he works with heavily weigh decisions made by intermediaries such as negotiating firm Coveware. Following Conti’s proclamation about Russia, Coveware stopped making payments to the group, said its co-founder, Bill Siegel. Coveware continued to negotiate with Conti, allowing time for the victim to assess the situation, prepare a public relations strategy and make arrangements to notify people affected by the breach.
=== unquote ===

More at the link.

Has 3xx reported anything to the FBI and OFAC??

Cheers,
Scott.

@Carlo – a bit more difficult, maybe, but still problematic for normal people I would think. Treasury.gov (from April 20):

=== quote ===
WASHINGTON – Today, the U.S. Department of the Treasury’s Office of Foreign Assets Control (OFAC) designated entities and individuals involved in attempts to evade sanctions imposed by the United States and its international partners on Russia. OFAC designated Russian commercial bank Transkapitalbank and a global network of more than 40 individuals and entities led by U.S.-designated Russian oligarch Konstantin Malofeyev, including organizations whose primary mission is to facilitate sanctions evasion for Russian entities. OFAC also designated companies operating in Russia’s virtual currency mining industry, reportedly the third largest in the world. This is the first time Treasury has designated a virtual currency mining company.

“Treasury can and will target those who evade, attempt to evade, or aid the evasion of U.S. sanctions against Russia, as they are helping support Putin’s brutal war of choice,” said Under Secretary for Terrorism and Financial Intelligence Brian E. Nelson. “The United States will work to ensure that the sanctions we have imposed, in close coordination with our international partners, degrade the Kremlin’s ability to project power and fund its invasion.”
=== unquote ===

The details matter, and IANAL. I do wonder, though, if any of the post-February-24 banking sanctions have had an impact on this saga with 3xx.

Cheers,
Scott.

Meanwhile, BoJo apparently wants to bring Imperial units back to the UK to short up his support with the mouth-breathers.

===

I’m an engineer to trade, if I want to tap a 6mm x 1mm thread, I take 1mm off 6 to get the drill size. (Diameter-pitch)

Same with 5×0.7, I need a 4.3mm drill, and so on.

To tap a 3/8 thread, I need a 37/64 drill, (because fuck you that’s why)

Imperial is useless. pic.twitter.com/wwU5WHrFoF

— Guffers (@gavmacn) May 29, 2022

===

Wait until you hear his rant about the eleventy-billion different plumbing fitting standards. :-/

Cheers,
Scott.

The story that 3xx fed us makes no sense at all. It’s analogous to claiming that hackers broke into the US Federal Reserve so as to rob savings accounts at some provincial bank. That’s idiotic. If you had control of the Fed, you’d aim a little higher, don’t you think?

I think we have to assume that 3xx has lied to their customers about everything.