Hey everybody,
In quotes below you will find the latest update from our site hosts. In case you haven’t been following along, our site hosts are the good guys, and “354” Data Denters are the ones who are stonewalling and withholding information. Our site hosts pass on what information they have from 354 Data Centers, which is usually meaningless word salad.
As a visual cue, you can pretty much assume that any text that refers to them as 365 Data Centers was written by someone besides me!
I’m sorry my message to you yesterday might have seemed confusing.
The confused mixture of phrases and updates from 365 sounds like word salad to me and has made it difficult for me to communicate clearly and effectively with you. I’ve done my best to accurately relay their communications to you.
I am fighting hard for your business. My primary focus has been to get our backups from 365. My secondary focus has been to create a temporary landing page system for your website so that you could communicate to your customers and audience.
Late last night I received some new information that I did not have before that has caused me to pause and consider taking legal action.
I’ve reached out to and started discussions with a law firm that specializes in corporate liability and class actions and we are in the process of initiating a class-action lawsuit against 365 Data Centers – for everyone’s benefit. Several customers in our blogging community have stepped up to co-lead the class-action lawsuit on your behalf.
I’ve been fighting hard for our clients. This has really gone on way too long and it’s beyond unacceptable. The damage being done to our brands is devastating.
I do expect to have the timeline from 365 this week and I do anticipate that timeline includes 365 Data Centers turning back on the cloud services and releasing our backups – at least that is what their team has repeatedly been telling me.
We do not yet know details of the new information that David found disturbing, but it’s surely nothing that would make us feel better about the situation. My hope is that we’ll be back up with Balloon Juice within a week or 10 days.
In the meantime, I’m trying to make this site as comfy as possible since our 3-hour tour has turned into a much longer one than that!
Crazy town.
I wish David was more forthcoming about the new information he received. But I’m not surprised that he and others are considering legal action.
Nice Gilligan’s Island reference. Humor always helps.
I still think aliens are involved.
And before anyone jumps all over me, I didn’t say that it was aliens that took down the blog. But it was probably aliens.
Why else all the secrecy?
What a cock-up.
That said, I actually like this new set-up and wouldn’t mind seeing it be made permanent. Comment text is a little light and hard to read on my phone, but laptop is fine.
So is Water girl Ginger or Maryann?
You’re doing yeoman’s work in getting this spritzy little holiday home up and running smoothly, WG. Very much appreciated in the absence of Teh Joose.
Oof. Well, that doesn’t sound encouraging.
So, what is the longer term plan? Once the backups are accessible, migrate to some other data center provider not run by a team of howler monkeys and bad-tempered badgers?
I like “354 Data Denters” in the OP, even if it was a typo.
My guess is the disturbing info he received was similar to that we saw from Reddit yesterday.
I wish there was a way to make the font contrast more with the background. Bold is fine but normal font is light. Hard to read for this ol geezer
@Baud: Watergirl is The Professor, making a blog out of two coconuts and some wire.
Thanks for all your work for us WG, JGC, and BizBudding.
The comments on Reddit seem (as I read them) to indicate that there was a ransomware attack and that may be why they’ve been so quiet about it. Not an excuse, and the attack vector seems to have been known for weeks/months so that doesn’t reflect well on 354 at all, especially their lack of communication, but there it is.
https://www.reddit.com/r/sysadmin/comments/urybn4/365_data_centers_ransomware_attack/ – Caveat Emptor.
Thanks again.
Cheers,
Scott.
As the anonymous person on Reddit who claims to be an employee of 354 data centers writes regarding the unknown, known root cause:
“ahaha…the full details of this are too embarrassing to share publicly, but basically it was a combination of an exploit and poor security practice.”
So if true, then yeah, it’s time to start the litigation.
ETA: Or what Another Scott said.
@Spanky
Perfect.
@baud
Let’s not get into the whole Ginger/Maryann thing. But I’m on team Maryann, for the record.
@Grumpy Old Railroader
Have you tried the Dark Mode button at the top right of the page? Looks like a white circle with a crescent moon next to it?
Maybe the dark background will make the text pop more for you.
Whatever David is hearing it’s not good.
3xx seem to be really pissing off their customers, at some point there’s going to be a flood of disgruntled ex employees spilling all
I do like this dark theme and the page header is a treat. Thanks WaterGirl
So I guess we’re all gonna be wearing coconut bras for a while?
I’ll get to work building a car out of bamboo.
This brings back memories of the day-long outage of 365, back in 2007 I think. A lot of San Francisco’s power had gone out due to a major fire at a transformer station, and their generators hadn’t been tested and simply didn’t work.
Meanwhile, the data center my company was in didn’t even have a blip. Or, rather, they had a blip, but it wasn’t one that affected any of their customers: one of their refrigerator-sized UPSes exploded from the initial huge power surge, sending the door flying over 20 feet. But it didn’t affect the power: the other UPSes worked fine, the fire extinguishers worked fine, and the diesel turbine generators worked fine.
IIRC Craigslist had moved from the data center we used to 365 less than two months before, and were rewarded with a day’s outage. This was, of course, before it was quite as common to have geographically-distributed failover servers.
@Tony Jay @WaterGirl
Dark mode is nice.
field expediency
This is what happens when you purchase your data storage from a circus clown in a darkened alley in a bad part of town at 2am.
@WaterGirl: much thanks to you for keeping this place alive. Like the Professor.
Not a huge dark mode fan, but it looks good on this site. In light mode, embiggenment had also been working for me.
Ooh, dark mode looks good. Although I think it will get on my nerves after a while.
Thanks for the update, and i also like dark mode – after I found the toggle atop the page. 😊
“Space Is the Place,” Sun Ra & the Arkhestra live
How about a GA elections thread? Early numbers are rolling in and Kemp is absolutely crushing Perdue in the governor’s race 73% -22% and Brad Raffensperger is running away from his MAGAt Trumper opponent 50% to 33%.
Will be interesting to see how this plays out in the general with the two most hated GOPers in the country from Trump’s point of view moving on to face Dems.
Also over in TX, Cisneros is very narrowly leading Cuellar by 52% to 48% in early returns.
I’m glad they’re going forward with a class action suit, but I hope they realize they’re unlikely to get much out of the company. Certainly not enough to compensate for any loss of business.
@Kent
I’m on Team Stacy of course, but those early returns are good news.
@Tony Jay “Have you tried dark mode . . . ”
Ahh, muchas gracias. El modo oscuro es mucho mejor
Still on paid suspension, haven’t heard nada, going into week 3.
Still don’t know what the complaint was, other than a generic “hostile workplace”,
I am suspecting that “employment lawyers” are involved, because the Corp has a long history of not covering my lunches and breaks,
So they are stuck deciding between kicking my ass to the curb on an unsubstantiated accusation* vs an employment lawsuit.
*(dumb thing, is despite the “confidentiality” of the complaint process, I know who filed it, and why. Because everybody in my Dept turns to me for advice, from dealing with Customers, to work arounds, etc, the Person, thinks I have “hire and fire”, and I don’t, and they are coasting and not getting up to speed.)
If we can keep dark mode once the normal site comes back I will be very happy.
The willow’s too close to the blog. Please fix.
Don’t know from nuthin’ but I’m wondering of the attack has been lurking in the datasets for weeks to months to years, compromising the backups as well. If that’s true, all it took was a trigger signal from a remote site to release the attack … which might be quickly resolved except for the compromised backups. Maybe 3XX is still trying to find out how far back the data compromise goes, and on which accounts it first appeared. That might explain the reluctance to release backups to any of their clients.
Back in the real world, a big Attagirl!!! for the tremendous work WG has done on the temp site, which is already beginning to feel homey. Thank you so, so much! I had no idea I relied on this site, and all the jackals, so much.
@Jay: wait, wut? Paid suspension?
Biden speaking. https://www.c-span.org/video/?520571-1/pres-biden-deadly-school-shooting-uvalde-texas&live
Interesting perspective on the blood-soaked war criminal Henry Kissinger:
https://twitter.com/KatyaYushchenko/status/1529106146832830466
I’m not clairvoyant but I think there will be another mass shooting (20+ dead) before the middle of June.
@Baud WaterGirl is Ginger and MaryAnn both! Plus a FYWP savant, it appears.
@Baud – I was afraid to ask that very question.
@Gin & Tonic
I cannot imagine why anyone would still be paying attention to anything that asshole has to say about foreign policy (or anything else).
Here’s a google drive link to a tarball (tgz) of the crawl of top level posts/comments (back to 2003) that I mentioned yesterday.
https://drive.google.com/file/d/10NwLTn1krg2ID0sd76xerP3JUFDji0XK/view?usp=sharing
I’ve downloaded it and tested (unpacked it) and verified that it’s the one I made.
Just in case it is needed. It could form an archive html site at least, though Carlo said that it is not nearly sufficient to rebuild the site.
@Elizabelle – good, obviously heart-felt comments from Biden. He laid out the issues clearly and stated it plainly that we will not forget those politicians who do not stand up to the gun lobbies and who block sensible gun laws.
I do wonder, though, how many more are going to die before the laws are actually changed. 🙁
Cheers,
Scott.
@Kent
If history is anything to go by, the Republicans will drop their internal disagreements the moment the primary is over. They may fight each other, but they hate the Democrats.
@prostratedragon: my son-in-law, a jazz drummer, plays with Sun Ra.
@zhena gogolia – I can’t use dark mode on sites. White text on black background makes my eyes throb.
@eclair,
Yurp, paid suspension. Employment law here. They have to pay me not to work while the “investigation” takes place.
Otherwise they would have to deal with an “employment” lawsuit, which could result in them having to pay me for up to a year, to not work, if the WCB Commission were to find that they didn’t have “grounds” or slanted the “investigation”.
By filing a “Workplace” complaint, and accepting it, the other employee hoisted the Corp on their own petard.
Mgmt doesn’t like me. I understand, I am a PITA when they pull their “punching down” bs. They could have just “redundant” me, but for that little problem of “who will fix stuff”.
Ran into this almost 2 years ago when the previous Store Mgmt also hated my guts. They reached out to my counterparts in other stores to see if they would do my job, part time, ( which they used to do, before I was hired), they said “no”.
@Bill Arnold – I appreciate Cole and WG wanting to keep all the old comments in the same dB as the current active site, but I wonder if it’s really worth the trouble. Without a good search system, trying to go back to find old comments/threads is very painful. And how often do people search for old comments anyway (beyond what comes up in a Google search, which typically can go back 10-15 years but no longer)?
Maybe this horrible incompetence by 354 can be an opportunity to think about re-architecting the next B-J so that, say, there is the current dB for the current year, and previous years are in their own individual annual dB. (I don’t know what one would do about comments that cross years – that’s a SMOP beyond my pay grade.)
If the active site weren’t dragging around 20-50 years of comments (that few if any ever look at again), then maybe future upgrades/HTML-plugin-changes will be easier.
Just a thought.
Cheers,
Scott.
@Josie
People pay attention to Kissinger because he’s a gray eminence of the foreign policy establishment. Those who are part of the blob can’t ignore him, because he’s the blobbiest of them all. Anything he says should be taken as the most conventional of conventional thinking.
@Jay: I am sorry to hear that, but I hope you are catching up on sleep, etc.
@Another Scott
I think the standard thing to do is to retire posts after they’ve reached a certain age or nobody has commented on them for some length of time. Retired posts can be converted to static HTML pages, which are way more efficient to serve than anything that requires hitting the database. Then all the elements can be purged from the database so it stays a sensible size and lookups are fast and efficient.
@Another Scott
The 20 years of data will be moved to an archive site right after we switch to threaded comments, like buttons, avatars and like buttons.
@Roger Moore – That makes sense to me. (I think comments are closed/locked on the original B-J site after 5-7 days or so.) Making older stuff static pages would seemingly make the dB much happier.
@WG – So you’re telling me there’s a chance!!
🙂
Cheers,
Scott.
Definitely Maryann and the Professor, not an ounce of Ginger in sight.
@eclaire,
Sleep, daydrinking, (because SWIMBO has the remote) the Hallmark Channel, cat scritches and naps, and job applications. I am way “underemployed”. Balloon Juice.
I used to run a high tech factory in Milwaukee, a Construction Company in Kamloops, now I fix broken tools for a Retail Corp and try to keep Customers who have no clue from cutting their legs off with a rental chainsaw.
@Jay: any idea when the “investigation” will be done?
This wouldn’t have happen if all of us were packing.
ETA: I really like the highlight carousel down below. Awesome!
@ cain
I think those would be super helpful on mobile!
@eclaire,
Based on convo’s in the past with others, ( really, they don’t think we talk and bi#ch with each other), another week and a bit to another week, a bit, and a month.
They are trying to figure out for the Corp, the Store, the AssMen, which is the better outcome, tossing me, or another Employment Lawsuit.
They also have to deal with the current employment environment. It’s taken “them” 2 years to staff us up to where we should have been 3 years ago. 2 of the staff should have been hired in Sept, so they would be “up to speed” by now, our peak season, but they finally filled the positions in January/February.
It’s frustrating. My DS is supposed to be in charge, but they are not, due to health issues. They do reports, clerical work, try to run interference, but they can’t lift, don’t know tools, and all too often, I have to run interference,
Anyway, holding pattern for now. It’s a crappy job but I have friends there.
how you feeling raven
@stacib: Well just hey-hey! Both current and past always had kickin’ drummers.
@WaterGirl: Actually, this is the most positive thing that I’ve read in this entire saga, and I understand now why you value your relationship with your site host. If this fellow is actually prepared to go to court against 3xx to get the backups — or, I dread to say, evidence that they actually have the backups that they claim that they have — then they not acting as passive cut-outs, and are OK by me.
This — levering the backups out of these dickheads — is totally the path. The “investigation” was a red herring all along. Maximum pressure on this one point is what is required.
@Jay: it always amazes me that higher ups don’t think the peons talk to each other and compare notes.
Hopefully this resolves quickly the way that you want it to. Til then, you have a busy schedule!
Bill Arnold: Oh, very nice. Thank you for doing this! Um, could I ask if you could, uh, make a smaller tarball, with just (let’s say) a month or so of data? If that’s easy? I’m curious to look inside, but …. don’t particularly want to download 6.6GB of data to do so. Only if it’s easy.
Like many, many others, I also noticed the (ahem) “gap” in community while B-J was down, and now that it’s back up, feel somehow that things are on the mend, even if (for the moment) the site is still hobbling along without full function.
And for sure, I appreciate what WaterGirl is doing to try to recover everything.
@G&T: Not looking for a shit-flinging argument here, and I will posit that HK should, at all costs, be kept away from all levers of power on account of his demonstrated propensities to abuse authority on behalf of his deformed and shrunken moral sense.
Nonetheless he has written some useful and interesting academic works on international relations that reflect some profound insights on how nations conduct their affairs. I feel that “Diplomacy” is a pretty good book, and “On China” is positively great.
It just goes to show that there are some people who will absolutely burn in hell, but will nevertheless leave behind something of value.
G&T: what a horrid person, bringing up receipts from the past! Why can’t she just let “Cambodia and Laos” Kissinger whitewash his past and trot merrily into the future, dispensing his ever-so-valuable and sage advice to the good and the great ?
ETA: I coulda worked in “whited sepulchre” if were only trying! Sigh.
I deal with everything from exploding UPs to overly pressurized fire systems on a regular basis. Farting hard drives in massive numbers, RAM with bad oxidation prevention strips. And reading any of this stuff for 365 they just sound like such amateurs. Things are going to fail just based on the number of parts there are in the cloud. How its handled and hopefully prevented the next time is what counts. Very little of it is about how we do business and more about how our suppliers do their business. Then again I guess we’ve never bought out someone else’s datacenters and integrated them, but we do lease more than enough space anywhere building new ones is complicated/slow.
I’d suggest we buy these guys out but its probably not worth the trouble, the affected customers are likely to migrate themselves to one of the big cloud providers after this mess. Incidents like this are why its hard to grow a cloud though. And why its a long way behind each rank on that ladder of providers.
Cole is a Skipper, Doug is Gilligan and I’m Thurston, of course
@David 🌈 ☘The Establishment☘🌈 Koch
Oh my god, Cole is most definitely the skipper!
The comments on the Reddit page only confirm what I suspected from the beginning! The only thing that answers ” why not provide our backup data?” is that everything was encrypted by the time they knew that they were hacked!
And from what I’ve learned from the IT media over the years, the only way to recover is to buy the encryption key from the bandits!
When we got our systems where we wanted them, nightly backup media was shipped to a branch office about 100 miles away from the HQ Data entered. I understand that the Cloud is different… but having duplicate DBs in remote sites seems to me to be the very definition Pof Cloud computing! Guess not for 365bullshit hosting!
@J R in WV
Our backup data was encrypted to begin with. Our backups were off-site, in two different geographic locations, two different states, actually.
That seems pretty secure until you find out that all of the locations together make up their data center, so when that got hacked, there was no access to anything.