Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Microsoft lost its keys, and the government got hacked (techcrunch.com)
317 points by tim_sw on July 18, 2023 | hide | past | favorite | 142 comments


I hate to go negative. I really do because, usually it doesn't improve the discourse whatsoever.

However, I feel like it needs to be said: Microsoft makes bad products[1]! They're overpriced, insecure, slow (it's astounding to me how slow their web properties are), and hard to use. Easily some of the worst UI I've ever seen and, what's worse, they've been like that my entire career. They keep slapping lipstick on the same pigs and claiming to have reinvented themselves.

Their support is so ungodly bad that it literally took one of their reps over a day to figure out what timezone I'm in. Meanwhile, two of my employees' emails were being randomly CC'd to the CEO for no apparent reason. The cause? Nobody knows. It just stopped happening. Even Microsoft couldn't tell me why it happened.

With a $2.5 trillion market cap, it may seem like they're killing it but, I really think they're riding on momentum just like IBM did. Twenty years ago, IBM's market cap was double what is is now and, in my experience, their products are every bit as bad (admittedly, I have almost no exposure to their big iron products).

So, what are our alternatives? I know many people who don't like Apple for various reasons. Linux on the desktop has been promised for 20 years. It's closer than it has ever been to being a solid replacement but I don't know that it's quite there yet.

It saddens me to say, I don't know what the answer is. I just know that we need to get away from Microsoft products and fast.

1: In my experience, the only two exceptions are VS Code and SQL Server.


After spending 5 years of corporate work having to wrestle with Azure (in a ci/cd capacity) I would never recommend Microsoft's cloud offerings to anyone.

Problem is that there is no other provider I would rather recommend. They are all bad in one way or another.


As a counterpoint:

I've worked with both Azure cloud and Azure DevOps (or whatever the feck they call it now) for a long time (maybe 10 years?), a lot in a corporate setting, but also for my startup.

Apart from the same price issues that exist across all the big 3 cloud providers, I actually really like Azure.


I love the conceptual framework and high level architecture of Azure: Resource Manager, sane naming compared to AWS, etc…

What I hate is the unfathomably sloppy implementation that their developers refuse to fix unless held at gun point.

You literally have to have an MS account manager about to miss out on millions of dollars to force a trivial fix through their pipeline.

They have zero “customer obsession.”


Actually I'd say they have customer obsession but that it's for a small minority of what We consider thier customers (to them the small fry including smbs are not thiet customers, they're just the chaff they have to deal with and control in order to properly get the wheat out of the whole).

BIG businesses and governmenta are thier customers, demanding maintenance of terrible legacy issues because they literally use bugs as features in thier even more poorly made and maintained custom systems built by some fly by night dev shop 2+ decades ago....

MS doesn't care about individuals, individuals are just eyeballs to them.


That's a myth. I've worked at those big government departments, and Microsoft ignores their requirements too!

Also, Azure is "all new", so legacy compatibility shouldn't be a big concern.

Except... it is, because a bunch of half-baked MVPs were thrown over the fence and now have to be supported while they work on the "proper" solution. We're up to the 3rd iteration of VM monitoring, the 3rd iteration of AAD client libraries, etc...


The big red issue with MS is they elaborately make it very difficult to integrate outside their ecosystem and they love holding customers hostage to ridiculous restrictions, but also Azure is very difficult to centrally manage things for all subscriptions. Others are as well but not as much. I actually like the azure UI and CLI as well as many if their offerings, but it's so difficult to piece together a bigger picture or make small changes without doing a ton of work and more MS products being involved.

Also,unrelated: Azure AD is now "Entra ID" (????) now lol.


Yeah, that Entra ID was the biggest "why?" moment I had in years. Seeing the new name no better or even worse than the old one I can already imagine the pain of renaming packages, namespaces, fixing documentation etc. Probably the most expensive marketing campaign with 0 gain I am aware of.


That’s easy and makes sense.

Azure AD was initially a shim to enable O365. Now it’s evolved into a pretty awesome identity platform for Enterprise and B2C.

The problem is M365 adoption (ie O365 + the other crap) is high. So they are out of stuff to sell. Entra will add the governance and compliance bullshit and increase revenue.

End of the day, Microsoft chisels $38-55/head for every corporate worker. Services are all about cash flow, so they are looking to grow that up to $100 user/month. Every $1 of subscription spend is like $400 on the scorecard for the bonuses of SVP staff.

Because of anti-trust, they can’t just do what Adobe does and raise the price every few months. They just keep adding services and move stuff around in the bundles. M365 Plus+ E9 with Entra technology will give you $350 in value for only $103/month!


Entra ID. As someone trying to get up to speed on Azure AD, suddenly all the recent tutorials I've watched are out of date. Very frustrating.

Same story when they moved to MSAL a few years ago.


AWS is pretty decent in my experience, and when managed with Terraform (which is all but a requirement if you don't want to go completely b0nkers) it's a breeze to manage.


And this is one place where I think Azure does better... I mean, the UI/UX sucks, but at least you can do everything in the UI without loosing too many brain cells. Google and AWS seem to require automation tooling to make sense, which IMO makes all of it harder to learn in practice. I tend to like learning some of those things visually first, then automate out of necessity as opposed to the other way around.

Not that I'm tethered to Azure... I've used a combination of dedicated server, DigitalOcean and Cloudflare for my personal stuff... though just started delving into the Cloudflare (edge compute) stuff.

There are definitely ups and downs to them all.


I'm not sure I agree completely. When I was learning Terraform, trying to figure out what the options a resource had and some of the possible values, I'd make the change in the AWS UI, then re-run Terraform to see what it wanted to change back. There weren't a whole lot of things in aWS that didn't have a UI to configure somewhere.


You can also configure a resource in AWS, configure the bare minimum in Terraform (i.e. required attributes only), then run a `terraform import` and then `terraform apply`.

(Terraform does suck in that its import functionality can't spit out the HCL representation)


Yeah the AWS console UI is far from incomprehensible. Though I think what trips a lot of people up is how the resources are often something you use to compose other higher level resources.


"Problem is that there is no other provider I would rather recommend. They are all bad in one way or another."

Could you explain a little further what you don't like about Azure and why you would still recommend it over the competitors?


Not the OP, but: it’s just sloppy. There is zero QA other than end-users bitching.

I recently collected some notes of bugs and errors for an MS rep. I was setting up just a .NET web app with a SQL back end, as simple and vanilla as it gets.

I found 60 bugs.

SIXTY!!!


I like the Azure bash cli... That's about the only thing I like from Azure.


I moved from Azure to AWS.

App Insights, DevOps pipelines, and WebApp slots look like future tech when compared to codepipelines and x-ray.

DevOps in particular feels so close to being a really great tool. Not having to jump between 8 different UIs to deploy something is magical. If only they'd polish the rough bits and invest a bit more into it.


We use DevOps for pipelines, but AWS for our prod cloud environment.


I'm hoping to push us that way, some hurdles with getting buy in from the whole team though.

That's why I wish DevOps made it a really clear cut choice. There's some really low hanging fruit and silly decisions that spoil the initial impression:

YAML pipelines have been the "official" option for at least 4 years now but when I create a new project, every issue has a "deployments" section that tells me it only supports classic releases. Classic pipelines/releases can now be disabled at the project level at least.

They're introducing a new widget that shows open pull requests across repositories but for an absolutely insane reason it's limited to 10 repositories, just why...

Environments still have no support for resources other than VMs and AKS.

The mandatory "sprints" backlog is just stupid as well. It's a clear artificial limitation rather than one necessitated by the platform.

Another issue is that Microsoft have done a poor job at communicating the potential longevity of the platform. The Github acquisition is the first thing that gets mentioned when DevOps is brought up as an option, which is a shame because DevOps feels much more capable/cohesive/advanced than GitHub.

Despite these problems it's still been the tool with the absolute least friction and most potential when it comes to organised development work. The alternatives I've used come nowhere close.


You mean the python-based `az` command?

running `az` without parameters takes 10 seconds on my machine. `az version` and `az --version` return differnt outputs and are also very very slow.

When I run `az account subscription enable` it tells me "Command group 'account subscription' is experimental and under development.".


> It saddens me to say, I don't know what the answer is.

I think the answer is a company willing to put in the work to force GPU manufacturers to provide high quality hardware drivers for a commercial Linux Distro. I mean all out partnership with the various vendors. Then sell prebuilts, maybe someone like System76 has come close, or has the capacity to do so. The Linux issues for me always seem to be hardware specific. Then its just making sure whatever the default DE is, has enough necessary polish.


I'm curious on the non-sequitur to GPU stuff? At large, running machine learning code has been easily doable in linux for a long time. Frustratingly hard, yes, but no harder than in Windows. Often more repeatable and cheaper to fan out to a fleet of machines, to boot.

Is there another aim of GPU that I'm ignoring?

Microsoft makes fine enterprise software, sadly. To know and believe that, though, you have to see what the alternatives are in the enterprise world. It is atrociously bad.

I do think Chromebooks are an interesting use case showing you don't need massive software. Schools have taken to those very well and are almost certainly competing with most enterprises for number of users and use cases.


> Microsoft makes fine enterprise software, sadly.

Hum, no they don't. They make shit and use anti-competitive practices to push every other offering out of the market.

The fact that you only see experimental alternatives without the maturity to get any polish is caused by them destroying the competition, not the reason they won.


I meant that as a purely "backhanded compliment." The bar is rather low in enterprise software.

That said, fully fair that, even in the field, they probably rely more on bad practices to shoulder out competition than I'm crediting.


I think it's because of game development. Many devs target Windows for DirectX, since it's easier and cheaper to target one platform, and Windows dominates desktops, so you won't get gamers to migrate without that support.

There are various ways to get Windows games working on Linux or Mac, but they aren't out of the box, which is significant enough to prevent wide scale adoption.


Steam has done an amazing job at attacking this. Though, large studios don't target linux largely because they have very little financial incentive to do so.

And it is more than installed userbase, which is very real. For the major distribution targets, the devkits have a ton of resources poured into them that major studios can't ignore.

That said, still feels like a non-sequitur? I can't imagine "gaming as a target" is a concern for cloud customers. Certainly not the government?


> Is there another aim of GPU that I'm ignoring?

I'm amused that you seem to have missed what the "G" in GPU stands for. As someone who occasionally plays games, I'm a bit salty about what it did to the prices.


But that seems unrelated to Microsoft in the corporate world?


I don't see why the discussion would be limited to the Microsoft corporate world. They make the standard consumer desktop OS for the world and that's still the standard for PC gaming. Personally, I'm thrilled with where System76/Pop!_OS and Steam/Proton are these days, and that's what I use, but I think that's still far from mainstream. The Steam Deck is probably doing the most for Linux based gaming these days.

Even if you think about the corporate world, I'd imagine that there's a lot of CAD on Windows and maybe some 3D animation still? I'm not that familiar with those uses.


Most heavy CAD industries are custom. Some likely still on commercial unix platforms.

Is a bit like movie firms. The popular view is that Apple has that locked down, but that seems to only be the prosumer market. Most of the main studios have custom render farms that are probably linux based, nowadays. Studios had the motivation to not get locked into large contracts that grow based on the size of their render farm. (Incidentally, this is why Wacom has such outstanding linux support.)

That said, fair point on this not being limited to corporate. Not sure why I had that bend from the intro post. I was probably biasing it more to the topic of cloud than I was corporate, but those seem roughly related?


people making massive income from the system exactly the way it is, do not care about doing this


> However, I feel like it needs to be said: Microsoft makes bad products[1]! They're overpriced, insecure, slow (it's astounding to me how slow their web properties are), and hard to use. Easily some of the worst UI I've ever seen and, what's worse, they've been like that my entire career. They keep slapping lipstick on the same pigs and claiming to have reinvented themselves.

Some think tank should make an analysis how many billions of dollars are lost in productivity across the globe if you compare the current MS Office stuff to an alternate of "the Windows 2000-era stuff, but with security patches" or "the FOSS office stack, with more money behind it".

Anyone wanna estimate how many digits that number has?

(Teams, while being ungodly slow, at least has the value proposition of being an integrated product - persistent chat, file sharing and voice / video calls - that really didn't exist in 2000)


MS SQL dev here, and I disagree because it's been going downhill in terms of code quality and especially documentation for a long while. Absolutely classic example why https://www.mssqltips.com/sqlservertip/3074/use-caution-with... but there are others


I have always believed that MS products are designed in a way to maximize the value of commercial support, certificates, etc.

It's walled garden for lack of a better time and solving problems easily and quickly dilutes the value of MSCE and the army of paid consultants that exist.

It is a completely different universe from Linux


I've had to get their certifications. They're just as much garbage as the rest of their offerings. The courses they provide are nothing but marketing material. They don't improve your understanding of the products at all.

I found out later that they use their certs as filters for resellers to help Microsoft identify which ones are serious players. What a waste of time and money.


> I found out later that they use their certs as filters for resellers to help Microsoft identify which ones are serious players.

If I understand what you are saying correctly, this is not exactly true.

The IT industry is a box-shifting industry, and therefore Microsoft, like all other vendors, differentiate their resellers primarily on dollar-volume.

So the REAL reason a Microsoft reseller is Gold is primarily because they have shifted more product than other resellers on lower tiers.

Sure, to actually get the Gold badge, you need to have X sales-reps who've passed the course and Y technicians who've passed that course. But bodies-with-certs are by comparison a very minor secondary criteria.

The key criteria is always that your company has sold $big_number per year. Your entire organisation can be hiring exclusively Microsoft certified people, but you ain't getting the badge unless you're shifting the product no matter how many certified people you have on staff.

And that is why Gold certified is meaningless. Because all it tells me as a customer is you've got good sales-reps on team.


It's even more hilarious because Microsoft are always loudly exclaiming telemetry is necessary for software improvement but based on how buggy and terrible most of their software is telemetry doesn't seem to be helping.


One anecdote I have about Microsoft is that I once set up in Hotmail my birthday 5 months in advance to what it was, probably to bypass some 18yo check. Eventually (5 months later, probably) I set it back to normal.

Since then, for a couple of years, my father got a notification saying that it was my birthday (and he even obliged to congratulate me, once; he was -- and luckily is -- old :) ). That setting was not available anymore anywhere, and my birthday was set to the correct date, but the problem still happened. I contacted Microsoft, but it was like they had no idea what I was talking about nor what could be the problem.

So yeah, I don't trust them nor like them either.


Linux on the desktop is here. Go all-in on KDE. Everything is seamless and just works.


> seamless and just works

Just say “better.” Gnome and KDE are decidedly not perfect, but they are both quite usable even by casual windows-only non-techies.

FWIW I’ve been using Linux as my primary desktop OS for more than 20 years—it does no one any good to oversell it. Touchpad scrolling rate is still broken on Gnome/Firefox. Angels have to sing (or use xorg) to make s3 suspend work. Fedora auto-update can break if there’s “too many” packages to update. Touchpad scrolling can be misinterpreted as swipe-left or swipe-right by KDE on some hardware.

The point is, if you bang against any of these sorts of issues in Windows, you’re frequently stymied. On Linux, you can either fix it, or at least route around the damage with an alternative.


My experiences with Linux have been anything but seamless.


Similar to my experience with Windows TBH.


yea same with windows having to open that command prompt to run random commands to fix issues


None of the comments here seem to address desktop OS in enterprise/government environment, which is what this article about. Linux desktop OS your personal computer? Sure. In an organization where most users are not technical, and where vendors do not release their software? Good luck.


> Sure. In an organization where most users are not technical, and where vendors do not release their software?

Most of these enterprise software (for non technical users) these days is just a web app so a vendor not releasing their software is not a problem here.


> In an organization where most users are not technical, and where vendors do not release their software?

Like schools? ChromeOS is Linux and seems to be doing ok.


Linux Mint Cinnamon is here as well. It’s more seamless than Windows. Unless you need to specifically run windows-only software. Still I’ll take dual booting or having separate machines rather than using windows as mu daily driver every day of the week.


This has nothing to do with large scale enterprise email security.


It does. Outlook is where it is for email because of Windows.


What does Outlook have to do with Windows? Outlook is a web app.

Outlook became popular because it was tied to Windows 20 years ago, but today this just isn't the case.


No, Outlook isn't a webapp in a lot of entrenched corporate set ups. Many are relying on outlook plugins that only work on the executable, or don't have Azure AD and use their own local domain controllers, or local mail servers, etc...

And besides, it's far easier to migrate between two webapps than it is away from a native app full of infra.


The only Linux desktop environment that I found that looks and feels integrated and polished is Gnome, but that is because it's a slow as molasses, corporate focused and dumbed down copycat of the other mainstream OSes.


> Everything is seamless and just works.

I have zero confidence in that testimonial, given my own experience with open source desktop environments, and KDE. zero confidence whatsoever.


If you really think about it from the point of view of defaults, Windows is a even worse mess.

* You often need to do non-standard system tweaks to get the system in a working state. It's everything-doesn't-work-by-default.

* You often need to install/prevent instalation of bad quality drivers and associated bundleware to make devices work.

* Microsoft can't decide what UI framework they will support (or drop support) so built-in applications happens to be built with the current "thing", the old ones staying on the older unsupported frameworks.

* You need to constantly tweak a combination of Windows 3.11 control applets, Windows 95 control panel applets, Windows 7 unified control panel glass, and Metro Settings Universal Windows Platform app to keep things working.

* Microsoft can't decide what random widgets go on the taskbar, or the file manager, or your start menu, so it's always disrupting things every few updates.

It is arriving at a point where the only "consistent" UI thing in Windows is Steam, and Steam isn't built-in, and is a hot mess of inconsistent UI itself.


I'm 12 years into Linux on the desktop. It's amazing. Converted my non computer friend to it. My partner freaked when windows 10 tried to install itself years ago, and she switched. I play triple A games on it via steam. Our TV PC runs it.

The experience is so much better I really don't get this mentality.


It's not a "mentality" - it's an opinion born from actual experience with broken Linux setups. I can find you literally hundreds of HN comments about all sorts of ways that their Linuxen have broken, and I can name a few dozen (possibly up to a hundred) myself.


Sounds like a skill issue tbh.


> With a $2.5 trillion market cap, it may seem like they're killing it but, I really think they're riding on momentum just like IBM did. Twenty years ago, IBM's market cap was double what is is now and, in my experience, their products are every bit as bad (admittedly, I have almost no exposure to their big iron products).

People are only not shitting on IBM so much anymore because they sold off Lotus Notes...


I don't see Lotus Notes as all that different than other Offices products developed by big Tech. I think it would be similar to Google's business platform with some more legacy automation but not unlike SalesForce or ServiceNow.

Having used Microsoft recently, after Google for work, I'm absolutely flabbergasted at how terrible Word, Teams, Outlook, etc are. I would describe the average Microsoft product as a zombie in its own feature set, at war with a different version of itself, and attempting to steal organs from other products to try to transition to every possible entirely different role in the suite.

Attempting to use a Microsoft product is like watching developers capitulate to some kind of bizarre internal political war between Product managers.


> So, what are our alternatives? I know many people who don't like Apple for various reasons. Linux on the desktop has been promised for 20 years. It's closer than it has ever been to being a solid replacement but I don't know that it's quite there yet.

When most people hear the word "Microsoft," their initial thoughts typically revolve around Windows and, at most, Office. However, it is important to note that a significant portion of Microsoft's profits actually comes from their Dynamics 365 offerings.


what does this have to do with the article, though? You just had a rant ready for when you saw a Microsoft-is-at-fault story came up?


It's unrelated to the article, and blatantly violates the HN guidelines:

> Eschew flamebait. Avoid generic tangents. Omit internet tropes.

https://news.ycombinator.com/newsguidelines.html

I flagged it - I suggest you do, too.


Microsoft exists to make money. If it made good products, it would not make money.


Good products are not required to make money.


I’d go with Gmail…


Their pricing has become bonkers over the last few years. Most people in traditional businesses need o365 for word/excel/powerpoint, and you effectively get email for free, so for many companies its a no brainer.


Until this happens


I work for NASA. Our budget is like $25B. We’re definitely inefficient and wasteful, but we still do a lot of new things. Just not nearly as much as we could.

Microsoft’s operating budget was like $123B last year. There are bugs in office products that have been there for years

What exactly are they doing with all that money?



MS is not "funding AI revolution", it is throwing a lot of money to OpenAI (which may as well be considered part of MS considering how much they owe it) to let it keep prices artificially low so users get addicted while it bleeds those money. Simultaneously trying to carve a moat to get some ROI on those money later.


you the user are not the customer, management and legal are the customer. You the user, are last, and least, in the priorities.


Based on my own dealings with Microsoft from such a management role over the years:

The only customer Microsoft seems to care about or is really even cognizant of its responsibilities towards, is Microsoft itself. Every single one of its products and services exists solely because and only for as long as some division of Microsoft believes it requires those specific things and whatever they're meant to evolve into over time, in order to manage its increasingly complex and ever expanding internal operations.

It's like a snake eating its own tail.


> What exactly are they doing with all that money?

Paying for ads and for developers that make ads I suppose.


[flagged]


Ah, no, but point taken. I can't think of an example myself because I don't know how Nasa functions.

Besides that, I think the valid critique (without the subtext) is, it's probably not a good comparison to measure a very large companies failings to a much smaller companies failings because their size is a definitively weighted factor in the outcome.

The caveat that I would add, is Nasa is publicly funded and it's probably ok to ask the question in the public interest. For private companies like Microsoft, just stop buying their products.


"Just stop buying their products" is silly answer when those products are epitome of corporate IT offerings. Especially where people doing "the buying" are completely different than people doing the support.

On the contrary - criticism should be loud, clear and persistent. Being private company does not make you immune to criticism.


This headline does a good job of pointing out why we shouldn't allow backdoors in encryption.


no one said it was a backdoor, though. you have to have a key to encrypt stuff, and it was that key which got leaked, somehow. a backdoor is a way around the use of a key to see secret messages, or an alternate key which allows you to view messages without compromising the key which encrypts messages in normal operation.

could have been a backdoor, sure, could have been something else. could have been carelessness. could have been blackmail. could have been a rogue employee. could have been malware, spear fishing, or any one of a hundred other things.

did I miss something or are you premature in blaming this on a backdoor?


If Microsoft can’t manage to hold onto these critical encryption keys, why do you think that <insert nation state x> will?

When your data is stolen because you were working in Cancun and the Mexican CIA was compromised, will you ever even know?

The point isn’t blaming a back door, it’s just pointing out that the more stakeholder that possess a key, the higher the probability of a compromise.


They didn't say it was an intentional backdoor. But if it were, the result would be the same.

It's hard to secure encryption keys. If you had a backdoor based on a key that you had to keep secret, leaks like these can happen. If Microsoft (or anyone) can't keep their encryption keys secure, they'd be equally unable to keep a backdoor key secure for the same reason.


But an encryption back door is either a second key that someone else holds or it is some piece of information the public should not know about.

Either way, when the details about the back door leak, the system ceases to retain the properties of secure encryption.


> no one said it was a backdoor, though.

That includes the comment you are replying to.


Wasn't there a time when Microsoft shared it's code with China govt on a locked computer in a locked room with access to only a few people as a demonstration that there product did not have backdoors for western countries to spy on China?


IIRC it wasn't just China, I think it was any government that required an audit of the Windows source code. I can't remember if the US government though requested access.


Any Windows product/system is a liability. I'm not sure how to put this lightly. It was designed to create a monopoly first before any other goal.

And it's not that the alternatives are perfect, but Windows is as secure as swiss cheese.


I know this is a bit of a thread jack, but this same attitude now playing out on Xbox in the console space. The recent FTC attempt to block the Activision merger surfaced emails saying that they were just going to outspend Sony to further consolidate and buy a higher position in the market, because they can.

There's nothing in any of Microsoft's revealed communications over the years where they ever seem to aspire to make better products. I mean, sure, obviously, there are pockets inside the company, but that has never seemed to be the purview of the board or the executives. If there's a thought leader in there somewhere who had passion to make great products like Jobs, I can't think of who it would be.

I almost feel bad for Microsoft, and I say this as a Microsoft-hater and Linux zealot who ran it on the desktop for 19 years before switching to Mac. It's the legacy backward-compatibility that makes Windows so attractive to a lot of organizations which makes it so vulnerable. OTOH, corporate IT divisions love it because Microsoft lets them do all sorts of stupid things to it, like prevent me from changing the desktop background, so it's kind of a deserved punishment.


> as secure as swiss cheese

In IT security jargon, the "Swiss cheese" model is a good practice: https://en.wikipedia.org/wiki/Swiss_cheese_model


> Any Windows product/system is a liability.

A self-evident realization: every single product/system is a liability.


this is not windows


Heartbleed did not affect windows.


Seems like they are assuming the same key was used to forge tokens and jump from that assumption to the conclusion that they found all instances of requests. If more keys were used to generate more tokens, isn’t it possible this attack had a much wider surface area?


As a rule of thumb, hacks are always worse than initially thought. See https://circles.page/5680a56b5c28af0998656e09/Hacks-worse-th...


Absolutely. Maybe Louisiana and Oregon DMV hacks?


One of the reasons why we moved away from using Microsoft products for our identity management to Okta.

Not that they are bad products per se, but as many organizations use Microsoft products they are prime targets for too many hackers and it’s hard going to bed thinking that your identity info might be hacked someday and/or not knowing if it will be.

The surface area that Azure/Microsoft have is just too large for it to assume to be protected by one company whose security as not been the most stellar. So we are actively moving to GCP as well.


Okta has been hacked a couple times now...


Was about to say, wasn't all their source code stolen too?

https://techcrunch.com/2022/12/22/okta-breach-source-code-gi...

Okta looked very bad during the entire saga where they kept denying they were hacked until the proof was insurmountable. Pretty sure there was a large discussion on HN at the time too.


Agree they have shortcomings too, I guess, what I was trying to say is its best not to put all your eggs in one basket and to spread them across different products/companies.


So, many weak links and no one technical is responsible with the macro vision?


To toot my own companies horn[0] we designed our authentication protocol OpenPubkey[1] to have two signers on tokens:

1. The IDP signer (like microsoft or google) 2. The Cosigner (like bastionzero.com)

...so that even if microsoft's signing key is stolen, the attacker also needs to compromise the cosigner's signing key as well. It's like multisig for authentication tokens.

I don't know if OpenPubkey would have helped in this particular case as the details are still coming out[2], but I think the future of authentication schemes must require that authentication tokens must be signed by multiple signers at different organizations; Authentication systems with single point of compromise signing keys is too fragile. Or put another way authentication via multiple independent roots of trust is just too powerful of a security tool not to use.

[0]: BastionZero, https://bastionzero.com

[1]: OpenPubkey: Augmenting OpenID Connect with User held Signing Keys, https://eprint.iacr.org/2023/296

[2]: It appears the key stolen was an MSA key, not an Azure AD signing key. The MSA architecture might not fit into the OpenPubkey model (or it might I don't know enough about MSA signing keys work to say). Had it been an Azure AD signing key then OpenPubkey would mitigate the theft of an Azure AD Signing key. https://www.microsoft.com/en-us/security/blog/2023/07/14/ana...


We use MS for some stuff and did use Okta for others. But... Okta fkd up and can never be used in any enterprise today. We migrated from it. Everybody should. Now we use MS and Google. Okta is probably the most over-priced service in history of CS especially considering the poor sec design. They do however solve the US-Franchise-Corp bs that I guess it was designed for.


Okta is also a prime target for hackers, and likely has a smaller security budget than Microsoft.


> Microsoft still doesn’t know — or want to share — how China-backed hackers stole a key that allowed them to stealthily break into dozens of email inboxes, including those belonging to several federal government agencies.


Speculation: insider threat


That would be my amateur guess as well which scares me in a couple of ways.


What's with all this fairly overt state sponsored hacking and everyone going "aww shucks" and moving on quietly?


War with China already started in the cyber space. In fact its probably the pre war intelligence gathering and backdoor installations


Isn't cloud fun?

Same keys to everyone's kingdom!



And yet, fascinatingly, very very little discussion. 60-ish comments was the most on any of those posts.


This really surprised me when I learned of this story on the weekend. Very little discussion, very little reception in IT news.

Someone hacked Azure AD[1] and accessed the data of 25+ orgs and the reaction is *crickets*?

[1] I'm not exactly sure which Azure component was hacked and the MS communication seems intentionally unclear and obtuse on this. It sounds like the private keys of Azure-internal auth servers were stolen, but technically those private keys were not supposed to work for the accounts/orgs in question, except they did, and, in any case, MS doesn't disclose how the private keys (one? multiple?) of their cloud auth servers were stolen.


There was one issue where bing search results could be editted for any query by random users with an AAD account, and made to include a script that sends the Azure credentials of the user that happened to use that query to god-knows-where. And it wasn't the only service that was practically open. And another great thing: there were no logs.

That's an amazingly big, world-wide security hole, and how much interest did it draw? Hey, look, there's a celebrity scandal over there.


This is not even the first time this year that Azure AD has been owned sideways.

See this mind-blowing blog post from Wiz where they live edited bing.com search results, injected XSS into O365, accessed exabytes of internal MS file storage and more: https://www.wiz.io/blog/azure-active-directory-bing-misconfi...

I have no idea how Azure maintains its reputation as a safe enterprise choice. Microsoft really just has an almost impenetrable reputation I guess?


I mean, we’re talking about key federal agencies involved as the impacted customer(s), too. Yet it’s like, eh, next story.


I saw the headline scroll by on my Ars Technica feed[1] but I admit I didn't bother to read it because there's like 10 major security breaches every single day. Just another one to add to the pile, there's more interesting things to read about.

[1] https://arstechnica.com/security/2023/07/microsoft-takes-pai...


Microsoft has published a more technical analysis[1] which was submitted in two of those HN stories mentioned above (which received no interest).

Microsoft have two identity services being MSA and AAD. MSA is used for consumer Microsoft accounts for use with products like Xbox. AAD is Azure AD that businesses use. When a client wants to authenticate to a Microsoft service, it asks MSA to sign a token for services accepting MSA tokens, or asks AAD to sign a token for services accepting AAD tokens. Someone has gained access to an "inactive" (but not "invalidated"/revoked) MSA signing key. It's not stated in the analysis whether Microsoft services such as Xbox would have accepted a token signed with an "inactive" MSA signing key, but it sounds like this may have been the case. It's just consumer privacy and security at stake and the attackers apparently weren't interested in that, so the analysis is rather ¯\_(ツ)_/¯ on consumer impact.

The problem for Microsoft is that they had incorrectly configured OWA API to accept tokens signed by the MSA identity services even when accessing mailboxes that should require an AAD signed token. So now the attacker could access mailboxes for all business and government users around the world (excluding GCC High, DoD, etc which Microsoft state are not exposed to public networks).

Microsoft only found out after one of their diligent customers queried Microsoft on why they were seeing logs for user mailboxes being accessed in unexpected ways. Speculatively (not in the analysis) this may have been noticed as a random IP address outside of allowed subnets accessing mailboxes at odd hours, and the customer checked their AAD configuration to ensure IP subnet restrictions were configured correctly and verified the user was not overseas.

The attacker looks to have been quite careless by accessing OWA API from very obvious red flag locations for legitimate users. For example, the attacker could have used a residential ISP subnet in Washington DC to access a GCC hosted mailbox of a government agency with a presence in Washington DC, and only done so in hours that an employee may be working from home in Washington DC. Or the attacker could have gone after soft targets like small to medium businesses that contract to the government, hoping these soft targets wouldn't be monitoring nor have the ear of Microsoft for prompt incident response. Instead, the attacker has used random dedicated hosting providers in places as far as Europe and accessed mailboxes during hours coinciding with business hours half way between the US and Europe. It's possible the attackers thought this attack would be detected quickly, and therefore were always just going for a hit-and-run approach knowing they'd get their data prior to the attack being noticed and blocked. But it does make the attacker look brazen at best, and unsophisticated and careless at worst.

[1] https://www.microsoft.com/en-us/security/blog/2023/07/14/ana...

[2] https://learn.microsoft.com/en-us/office365/servicedescripti...


As an update, the attackers were discovered by the customer due to mailbox access event logs showing an unexpected mail client to access mailbox items.[1][2] The attacker should have spoofed a realistic mail client.

[1] https://www.cisa.gov/news-events/cybersecurity-advisories/aa...

[2] https://learn.microsoft.com/en-us/office365/servicedescripti...


Wow. It's hard to say which is worse "our internal apps to control 1P MS properties were marked multi-tenant and did no real AuthZ"... or... "we leaked an MSA signing key and had our token validation so f**ked that (presumably) any signing key was valid for any key-under-test, regardless of expected configuration".

Just stunningly bad and would make me reconsider how much MSFT I hold, except that no one seems to care.


Yeah, this is really, really weird.


Mentioning China negatively is the kiss of death.


On HN? I don’t see censored comments with China mentions and this thread has been visible yet very few comments. What sort of manipulation I’ve seen on HN is threads being flagged or drop from top 30 to top 240 within an hour. Im suspect intel agencies have some ability to tilt the scales here on HN.


I feel I can't discuss some foreign nations anymore in a critical way since I fear giving mentally unstable war mongers ideas.


What do you mean? I can't say I have ever seen war mongering on HN.


Ye true, I meant in general outside of HN. How news articles are used to push agendas etc. Like, I got this thing in the back of my mind reading about social injustice in some place that it is some push to build up pressure to bomb them.


Just posting these links, some of which have no comments, is a bit much and not actually helpful. It would be better to do a little more curation and choose the most active ones.


Played with writing a little script to format/sort that:

Microsoft takes pains to obscure role in 0-days that caused email breach - https://news.ycombinator.com/item?id=36740133 - (64 comments)

Microsoft still unsure how hackers stole Azure AD signing key - https://news.ycombinator.com/item?id=36731731 - (9 comments)

WSJ: China allegedly hacked Microsoft Azure - https://news.ycombinator.com/item?id=36693469 - (6 comments)

Microsoft Warns That a Chinese Cyberattack Breached Government Email Accounts - https://news.ycombinator.com/item?id=36697272 - (5 comments)

Thousands of Microsoft Customers May Have Been Victims of Hack Tied to China - https://news.ycombinator.com/item?id=36692494 - (3 comments)

Microsoft government email compromised (and quietly fixed) - https://news.ycombinator.com/item?id=36696852 - (3 comments)

WSJ: China Hacking Undetectable from Less Expensive Microsoft Services - https://news.ycombinator.com/item?id=36713560 - (2 comments)

Chinese Hackers Breached Government Email Accounts, Microsoft Says - https://news.ycombinator.com/item?id=36690598 - (1 comments)

China-based hackers breached US gov email accounts, say Microsoft, White House - https://news.ycombinator.com/item?id=36694188 - (1 comments)

Microsoft mitigates 0-day, allowing forged AD tokens - https://news.ycombinator.com/item?id=36708294 - (1 comments)

A cloud flaw gave Chinese spies a key to Microsoft’s kingdom - https://news.ycombinator.com/item?id=36716028 - (1 comments)

Chinese hackers breach email of Commerce SEC Raimondo and State Dept officials - https://news.ycombinator.com/item?id=36719412 - (1 comments)

13 posts with no comments: https://gist.github.com/bcjordan/b579960792ee146f905c3e66f39...


"Use MS products and you will get hacked" is not exactly news, is it ?


How did you even get all those so fast?


Search box, typed in Microsoft and scrolled through a week's worth of submissions.


Look at hn.algolia.com, they index pretty much everything on here.


Perhaps preceding your list of related links you could state that the following is a list of (recent?) related HN submissions.


I'll consider it.


It’s implied imo


Yeah, the submission IDs are all very close.


You thought perhaps they commented a list of random unrelated HN submissions?


The links are all just https://news.ycombinator.com/item?id=########. It's obvious they refer to some content on HN, but it could be comments, replies to comments, etc.

By adding a small bit of human text, the human doesn't have to ponder the list and wonder what they will get if they click a link.

And if there were human text, someone arriving by way of the parent comment link would have an easy time understanding the context without having to scan a lot of characters and make assumptions.


Did anyone start getting Microsoft "single use code" emails last night?

(Maybe if you have an Microsoft consumer account that you don't use, forgot you had, and might be incompletely set up.)


Interesting that it aligned with their AAD rebrand.


What do you expect? Microsoft is a gaming company. They don’t care about security.


This sounds like an inside job.


Speculation. This type of key issuance is given to FBI and NSA and other partnerships Microsoft has. China hacked their way up to get the keys.


American companies are generally forbidden by EEOC rules from having reasonable security precautions unless some product is associated with a government contract and can require full blown security clearances. As a result you can safely assume that any given department with a juicy portfolio is fully compromised by foreign intelligence.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: