Perfect. So if I'm understanding you correctly, I can throw together a game with arbitrary/irritating rules that many people won't like, and I'll just call it an autism simulator.
Feel free to go back to a world in which human beings stopped developing technology at the sharpened stick. It's pretty easy to buy a large, remote plot of land in many states in the US, so you can live in your technology-free paradise.
Just to clarify, this statement will always be true: "N members of our species aren't ready for technology Y". And N will always be greater than or equal to 1.
So I can enjoy a completely isolated life until the bomb takes me out, and the only interaction I need to have with anyone is when it's time to pay my taxes? I'm not sure that's a serious alternative at all
Since you used the term "shitty gatekeeping": to me, your comment reads like the most generic kind of optimism you see everywhere on the internet about everything ever. Shitty optimism.
No one told you weren't allowed to be excited, but you took it that way anyway.
The amount of self-aggrandizing and lack of self awareness tells me this author is doing to do all of this again. This post could be summed up with "I should have had backups. Lesson learned", but instead they deflect to whining about how their local desktop is a mess and they NEED to store everything remotely to stay organized.
They're going to dazzle you with all of their hardened bunker this, and multiple escape route that, not realizing all of their complex machinery is metaphorically running off of a machine with no battery backup. One power outage and POOF!
The author doesn't grasp what putting all your eggs into one basket means:
> Before anyone says “you put all your eggs in one basket,” let me be clear: I didn’t. I put them in one provider, with what should have been bulletproof redundancy:
That's one basket. A single point of failure. "But it should have been impossible to fail!" Backups are to handle the "impossible" failure (in reality nothing is 100% reliable).
This one time, traveling through Asia, a simple merchant transaction triggered a fraud alarm on my card. The default for my bank at the time was to cancel my card automatically. This was before the days where cards could become unblocked. I had to travel to another city to pick up a new card in 10 working ways. This was a Mastercard credit card. I thought I was smart traveling with both a mastercard and a Visa card. Well, the Visa card was automatically cancelled too. Due to the same event. No cards for me to use to get to that city and I had to resort to a dodgy western union transfer to move forward. Also, try booking a flight with cash, it’s not fun.
My point is that the basket that eggs are put in is not always clear in hindsight. I wasn’t even aware that Mastercard and visa shared fraud alerts and that they were automatically linked.
The author’s article is not about backups, it’s about accountability.
My post is not about backup strategy, it’s about what happens when the infrastructure itself becomes hostile, and support throw you from one team to another. AWS didn't just delete files.
They gaslit me for 20 days while violating their own stated policies.
Visa and Mastercard are also an excellent example of two companies we trust way too much despite having proven many times they don't deserve that trust.
Fully agree. It's the same reason why you wouldn't put a repo on Github and then mirror that repo to Github. At a minimum any good mirror would be on a different provider, and ideally (if we get real picky) on a completely different cloud service.
> I’d done everything right. Vault encryption keys stored separately from my main infrastructure. Defense in depth. Zero trust architecture. The works.
Did you? Is putting all your eggs in one basket "defense in depth"? Is total trust in AWS "zero trust architecture"?
I'm not defending AWS here; they fully deserve all the fallout they can get from this, and I do feel for the dev who lost all their stuff through AWS's fuckup. Lots of people do the same.
My current employer does the same. It's a major bank, and all of their stuff is Microsoft. Azure, SharePoint, Office, Teams, the works. I think it's foolish to trust a single foreign company with all your vital data and infrastructure, operating in q country where the government demands access to everything, but this is what everybody does now.
We trust "the cloud" way too much, and expose ourselves to these sort of fuckups.
I dont disagree with your broader point—centralizing everything in one provider is a systemic risk.
The architecture was built assuming infrastructure within AWS might fail. What I didn’t plan for was the provider itself turning hostile, skipping their own retention policy, and treating verification as a deletion trigger.
> The architecture was built assuming infrastructure within AWS might fail.
From what i gather it was not. Or did you have a strategy for a 0-warning complete AWS service closure? Just imagine AWS closing their doors from one day to the next due to economic losses, or due to judicial inquiries into their illegal practices: were you really prepared for their failure?
The cloud was never data living in tiny rain droplets and swimming across the earth to our clients. The cloud was always somebody else's computer(s) that they control, and we don't. I'm sorry you learnt that lesson the hard way.
It probably wasn't even hostility; it's just that accounts is also an infrastructure component. And when that fails, everything fails. Tying everything to a single account creates a single point of failure.
It's one of the reasons I don't use my Google account for everything (another is that I don't want them to know everything about me), and I strongly dislike Google's and Microsoft's attempts to force their accounts on me for everything.
If you have very important data for you, and you don't pay very high bills to AWS, you should really have at least a cold backup somewhere else (even on your own hardware).
If you have a big cloud account, paying big money every month, at least with AWS you are in a pretty safe spot, even if people will say a different thing here.
And if you have a similar horror story with a tens/hundred of thousands of dollars (or more) monthly AWS invoice, please speak, I'm very curious to learn what happened.
I'll concede that I'm hugely empathetic for people that suffer data loss. The pithy aphorism about there being two types of people -- those who haven't lost data, and those who do backups -- is doubly droll because only the second group really appreciates the phrase.
But it's surprising to find people with more than a decade in IT who don't appreciate the risks here.
The timeline reveals there were 13 days from when the first signs of trouble surfaced, to when the account was deleted. So a fortnight of very unsubtle reminders to do something AND a fortnight in which to act.
(I recently learned the phrase BATNA[0] and in modern <sic> IT where it's Turtles as a Service, all the way down, it's amazing how often this concept is applicable.)
Author seems very keen to blame his part-time sysadmin rather than his systems architect. I can understand the appeal of that blame distribution algorithm, but it's nonetheless misguided.
The phrasing:
> But here’s the dilemma they’ve created: What if you have petabytes of data? How do you backup a backup?
inverts the horse & cart. If you have a petabyte of data that's important, that you can't recreate from other sources, your concern is how to keep your data safe.
If you're paying someone to keep a copy, pay (at least one other) person to keep another copy. Even that isn't something I'd call safe though.
> Me: “You’re answering like I’m Piers Morgan asking ‘Do you condemn October 7th?’ and you reply with historical complexity dating to 1948.”
Yeah...
If I'm working tickets at AWS that kind of dickishness is going to ensure that I don't do more than the least amount of effort for you.
Maybe I could burn my entire weekend trying to see if I can rescue your data... or maybe I'm going to do nothing more than strictly follow procedure and let my boss know that I tried...
I did have backups. Multi-region. Redundant. I followed AWS’s own best practices to the letter.
The only failure I didn’t plan for? AWS becoming the failure.
The provider nuking everything in violation of their own retention policies. That’s not a backup problem, that is a provider trust problem.
The reason i did not kept a local copy, was that i formatted my computer after a hardware failure, after the nurse dropped the laptop in the hospital i was on. Since i have a AWS backup, i just started with a fresh OS while waiting to get discharged to return home and redownload everything.
When i returned 6 days days later, the backup was gone.
As someone who has lost data myself, i'm really sorry this happened to you. I refrained from commenting on your article originally, but you seem stuck in a mental state blaming AWS for deleting your "backups" that you established with "best practices".
But you need to be aware that you never had backups in the way most sysadmins mean. If i need a friend to take care of a loved one while i'm away, and my backup plan is having the same person take them care of them but in a different house or with a different haircut, that's no backup plan: that's bus factor = 1.
Backups mean having a second (or third, etc) copy of your data stored with a 3rd party. Backup assumes you have an original copy of the entirety of the data to begin with.
From this point, and i'm sorry it bit you like this, but you never followed any good sysadmin practices about backups and disaster recovery. I have no idea what AWS best practices say, but trusting a single actor (whether hardware manufacturer or services provider) with all your data has always been against the 3-2-1 golden rule of backups and what happened to you was inevitable.
Blame AWS all you want, but Google does exactly the same thing all the time, deleting 15-years-old accounts with all associated data with no recourse. Some of us thought the cloud was safe and had all their "cross-region" backups burn in flames in OVH Strasbourg.
We could never trust cloud companies, and some of us never did. I never trusted AWS with my data, and i'm sorry you made that mistake, but you may also take the opportunity to learn how to handle backups properly in the future and never trust a single egg basket, or whatever metaphor is more appropriate.
You find it interesting that people that play the exact same role on both teams can be rivals?
By your logic golfers could never be rivals, because they're never interacting or defending against one another -- they're just playing against the course.
> there are people on both teams who really are tackling those guys.
This is the critical distinction.
Why is the commentator talking about Brady vs Peyton, when actually Peyton is doing nothing to stop Brady getting touchdowns?
Why don't they focus on whoever is making the decisions on the defensive side? That would seem to matter more as a struggle between combatants. It's not like each defense is some nameless, generic unit that simply reveals who is the better QB.
There might be some depth to the attack vs defense strategy that is worth highlighting, above the QB v QB.
You say that like caring about the situation I find myself and my son in, not liking it, and sharing that frustrating reality with others is a bad thing.
Please don't reply to a bad comment with an even worse comment. That's the epitome of inflammatory behavior. It's not what this site is for, and destroys what it is for.
I apologize. I think if you review my comment history, it is apparent that I follow the guidelines, mostly. I flagged the comment, but I felt particularly inflamed in the moment. Personally, I feel the comments are of equal quality.
I appreciate the apology but not the equivocation. Addressing someone like that is never ok and means you surrender the right to criticize someone else. If we want others to be better we need to hold ourselves to a high standard first.
I've only seen this now, three days later. Sorry you felt unfairly attacked by my comment. It just seemed you were doubling down on the view that your comment being equivalent to the one you were replying to, which, is just not how we want to think about things (comment should be evaluated in their own right).
Maybe I was too harsh. We had a quite a lot of politics-related flamewar last week, which always makes the job less enjoyable, most-of-all when users take out their political anger on us.
Sorry if I seemed to lack empathy. I did appreciate the apology and I hope you'll keep using HN and making an effort to contribute positively. And thanks for sharing your feelings about my comment, it's helpful to know, so I can better calibrate in future.
You are really this shocked that some competitions ban certain methods of achieving the goal? Are you shocked that traditional wood chopping competitions ban chainsaws?
No, my surprise has nothing to do with fairness and everything to do with how unlikely I feel it is that circulated ribs would hold up to ribs smoked end-to-end. But people downthread with more information disagree, so...
