> the world owes Andres unlimited free beer. He just saved everybody’s arse in his spare time. I am not joking.
Agreed.
> and face significant amounts of online abuse from users of the software.
Much as I’d like this to change, I suspect it will not. I’ve been doing open-source work since the mid-90s, and have suffered abuse that would curl your toes, but I still do it, for reasons that would be obscure, for many folks, hereabout.
I think the onus is on the users of the software. It’s currently way too easy to make “LEGO block” software, by splicing together components, written by others, that we do not understand (which, in itself, is not necessarily bad. We’ve been working this way for centuries). I remember reading a report that listed YAML as one of the top programming languages.
If companies insist on always taking the easy way out, they continue to introduce a significant amount of risk. Since it’s currently possible to make huge buckets of money, by writing crap, we’re not incentivizing high-Quality engineering.
I don’t think there’s any easy answer, as a lot of the most effective solutions are cultural, as opposed to technical. Yesterday, someone posted a part of an old speech, by Hyman Rickover[0], that speaks to this.
If companies insist on always taking the easy way out, they continue to introduce a significant amount of risk.
This, each company needs to take some amount of responsibility for the stack that they use. A company that I worked for, sponsored upstream maintainers, sometimes for implementing specific functionality, sometimes without any specific goal. If more companies did this for their open source stacks, open source development would be much better funded.
Of course, it will always be hard to detect very sophisticated attacks from actors that can play the long game. But if maintenance is not just a hobby, PRs can receive more attention, and if it's not an added workload besides a day job, there is less churn/burn-out which can be used for malicious actors to take over a project.
"...each company needs to take some amount of responsibility for the stack that they use. A company that I worked for, sponsored upstream maintainers, sometimes for implementing specific functionality, sometimes without any specific goal. If more companies did this for their open source stacks, open source development would be much better funded."
Agreed, 100%. And yet: In a comment on Slashdot one guy said that his company had "thousands" of machines running Linux, and he was "proud" to have never paid a penny for it. I called him a parasite: open-source brings an immense value to his company, and he should support it.
My comment got a lot of hate, which I just don't understand. Sure, OSS licenses say you can do whatever you want. However, there is surely some ethical obligation to support something that your entire business depends on?
It sounds to me like that person was at least helping to legitimize and popularize the software they use. Open source needs users just as bad as it needs contributors to be successful. Linux would have died if it never had large scale adoption by "parasites".
Most open source contributors aren't doing it to be paid; if there isn't enough demand or support they simply stop working on it and that's it. Usually once a project is big enough then there's enough demand from users that big companies start donating both money and employee time towards a project (AWS for example had dedicated employees contributing to Redis, and Linux as a foundation is supported by countless companies). The emphasis is on the money going towards supporting the project to keep it sustained, not maximizing profit.
> Usually once a project is big enough then there's enough demand from users that big companies start donating both money and employee time towards a project
I don't agree with “usually” there, I think it is an exception rather than the rule.
Where it happens, it tends to be higher profile, top-level, visible, projects that get this treatment from the commercial bodies that rely on them. The smaller projects that they might depend upon are likely to stay less visible, less thanked, and unsupported by those using them (directly or indirectly). What has happened with xzutils is a very good example of this and the potential dangers the situation poses.
I don't know how we should address this. It certainly isn't the responsibility of people like Collin to address, unless of course they want to take on that responsibility.
> if there isn't enough demand or support they simply stop working on it and that's it
Sometimes even if there is enough demand they stop, as is their right, when other priorities come up in their life (or they simply lose interest). A “community” using their stuff does not, and should not, automatically make them beholden to that community.
I have been paid to write FOSS, I have paid others to do so, I have dragged cheques out of major banks to sponsor conferences. And yet I still do not know of a convincing way to get money from where it sits to the pockets of all the devs working on the stack, let alone in a fair manner, let alone in a way to allow the vibrant and high quality development processes that exist (ie “the government pays developers” will almost certainly be a dead hand on development)
Honestly the problem feels
Like it is capitalism itself … which is weird.
It is. You don’t spend money if you don’t have to unless you are motivated by something that isn’t ‘more money‘. E.g. taxes or charity (which let’s face it is usually taxes).
Thus if the license says ‘you don’t have to pay money to use this, nor do you have to do anything at all’ it’s no wonder it gets used for nothing in return. Stallman was obviously right with the GPL keeping the source free instead of end users.
Why do you mention capitalism and not economy in general ? Any resource allocation exercise is "complex" and "hard". Probably there is a better way but I expect any progress to be slow. The open source movement started in the 80's and it took a long time to become (a bit more) mainstream.
I have asked that question many times [1] when confronted with this type of muhhh capitalism remark and have yet to receive a valid response. I suspect it is the result of the ideological lopsided makeup of educational institutions which churn out the message that capitalism bad, ${my_ideology} good where ${my_ideology} tends to stand for one on the left side of the political spectrum. This does not provide students with the means to actually defend their position, only with the correct slogans to use - a rather shallow base for a political conviction.
Of course HN is not the place to discuss politics so in some way it is probably for the best that these questions remain unanswered. It would be better still for those muhhh capitalism - and for that matter muhhh whatever-ism - remarks to remain unwritten since they are not supposed to be discussed here anyway.
> And yet I still do not know of a convincing way to get money from where it sits to the pockets of all the devs working on the stack
This is the purpose of licensing and you're describing a failure in the popular licenses that were popularized by the FSF and similar groups.
We have been indoctrinated. It is not always appropriate to give away your source code for free to everyone, and corporations without the moral qualms that individual developers have are happy to take advantage of their naive licensing.
We, as developers, need licenses that are less liberal, such that when corporations use our code, we are properly compensated for the value we have provided to them.
I think this means a deep rethinking of how we license software, and reopening ourselves to the possibility that proprietary licensing is not always bad.
OpenBSD's famously crusty owner along with his leads short-circuit this sort of entitlement by asking "If you have a complaint, where's your commit to fix it?"
Honestly, I loathe that attitude. Maybe OpenBSD has such a limited audience that the overlap between "people who discover problems" and "people who can fix problems" is perfect, but generally it's not.
To add, there's a massive difference between legitimate criticism and ungrateful entitlement. Complaints, even bad ones, are a crucial part of the feedback loop. If 99% of users say "this is terrible" with no elaboration, sure it doesn't tell you much, but it sure as shit means that there is probably something wrong that needs to be fixed.
Or it could mean that the 99%, or a large part thereof, have chosen the wrong tool for their task, which is a problem for them to fix not the tool's maker.
> This, each company needs to take some amount of responsibility for the stack that they use. A company that I worked for, sponsored upstream maintainers, sometimes for implementing specific functionality, sometimes without any specific goal. If more companies did this for their open source stacks, open source development would be much better funded.
This is good but I don’t think it’s enough to cover stuff like this which is deep in the stack and probably doesn’t have enough demand to warrant the trouble of direct funding (thinking of how many organizations make it easier to buy a $50k support contract than a $50 open source donation). I’ve been wondering if you could get companies to pay to something like the Linux Foundation, Apache, etc. for a general maintainer group tasked with basically taking something like Debian’s popcon and going down the list of things which aren’t well-established projects to pick up maintenance work.
One of the problems with the "money solution" in this case is that xz is a very small, relatively stable software. Sure things like the linux kernel, firefox, gnome or openssh could use huge donations to fund multiple developer day jobs for years.
But xz is small, it doesn't need to add a lot of new features constantly. It does need maintenance, but probably only like a couple hours each month – surely not enough to warrant a full-time day job. So what does the dev do to spend the other 90% of his time (and earn the other 90% of the money)? Some people don't like juggling multiple jobs (very stressy), some corporate jobs don't allow it, plus you've done nothing to reduce the Bus-factor (ideally any vital library should have 2,3,… people working on it, but who can take of just 5% of his day job to devote to open source maintenance?)
You could have a "Foundation" funded by corporate stakeholders. It doesn't even have to have a scope as broad as all open source software; it could be a "Linux Foundation". Google and Amazon and Netflix and whomever else uses Linux would be members of this Foundation on some kind of tiered plan, and their membership fees would go to maintaining critical, Linux-adjacent software.
While I’m not specifically opposed to this, you have to be careful about the edge cases and how they end up being rated. SQLite is a great example for the variables you started with:
> Size of team
3, with no outside contributions accepted into the core.
> The language
C
Those two, presumably, would end up being heavily weighted towards having a low score.
> Testing methodology
Here they’d probably get some points back based on their ~10:1 test lines vs code ratio.
Just because the kidz doan' like it, does not mean it's bad.
1 is the loneliest number (have to be an old fart to get that), but big teams can be a problem (Camel is a horse designed by committee). I'd say the experience and track record of the principals are a big factor (see "Linux" and "Git").
i think you have it backwards. giant teams == big risks, esp. if there isn`t strict and formal processes to eval the new contributors
3 people are easy to evaluate, have been on the project forever, and presumably know what they`re doing.
C shouldn`t be a problem per se -- again, it`s been around a lot, and while it makes it easy to bork pointers, it`s not a meme language that we don`t know much about
I think the 'easy' answer is liability, same as it is for any other complex human engineering achievement. Liability though would mean at the very least allowing commit access to only identified individuals and companies that are willing to pay for insurance coverage (to gain commit access).
This would probably ruffle too many feathers from the GNU old-timers, but I really dont see any other option. We are way past the tinkering-in-the-basement days of Linux/BSD hackers when most of us just wanted a cheap Unix box to play around with or to avoid Windows. A massive percentage of the civilian (and other) infrastructure is built on the shoulders of unpaid hobbyists. There is already massive liability at the social and corporate level. Time to deal with it.
EDIT: Ok, sounds like I have to describe this better: 1) you (governments) force commercial providers to assume liability for security issues and exploits and force disclosure, etc, 2) their insurance premiums go up, 3) to reduce premiums they only use checked/secured software, 4) that means maintainers of at least the critical pieces of software get paid via the (new) channel of risk reduction. Doesnt apply to all OSS, doesnt even apply to all distros. But it creates an audit trail and potentially actual compensation for maintainers.
Sounds like a good way to kill off open source entirely. This is luckily unlikely to happen.
As for throwing money at the maintainers, honestly, it’s complicated. A lot of people aren’t doing open source work for the money. Money too often comes with strings, requirements to prioritize what the funder wants to prioritize, pressure to perform consistently, it becomes an actual job.
Not only does this turn off a lot of the types of folks who make the best contributions to these projects, but it bends the priorities toward what would make the most money for the funder. And as this article points out, real security investments often fall by the wayside when profit is involved.
So yes, companies should encourage their workers to contribute to these projects, donate money to the foundations that fund them, hire important maintainers and give them carte blanche to work on open source. But we have to be careful. Making it all completely transactional is directly contradictory to what drives a lot of the contributions.
Given the level of age discrimination in software engineering, maybe we should add a stipend to the pension plans of retired developers who work on open-source projects.
Yes, the devil is in the details, but I think the basic concept is worth exploring
At least two of the maintainers of the framework I originally authored (and is now being maintained by a team, and used worldwide), are retired engineers. They are outstanding engineers, with great pedigrees, and bring real technical leadership to the project.
In that particular project, we're all just Paying It Back (not forward), but other projects could likely benefit from the participation of Grumpy Old Farts.
We are not talking about all of open source here; there are crucial bits of code and less crucial bits of code. LZ/OpenSSH was obviously in the first category. How do you determine which ones are more critical? same as you would for a bridge or a plane: by risk, impact, etc. That's basically liability.
And obviously a non-insured piece of code that assumes no liability whatsoever can still be free and maintained via IRC, same as it ever was. I dont see how this "kills all open source".
Companies and in fact everyone has the choice to NOT use software that comes without warranty. But, of course, the cost difference will be astronomical. Alternatively companies and everyone have the choice to inspect open source software for security problems BEFORE use. Of course, astronomical cost.
This is an attempt to shift costs onto open source developers. Which, aside from being totally unjust, won't work. There's a legal expression "you can't squeeze blood out of a stone". Shifting costs onto people that can't carry those costs doesn't work for the same reason supporting a skyscraper with a toothpick doesn't work. The toothpick breaks, and when the skyscraper lies collapsed on the floor, nobody blames the toothpick. Hell, they might say the toothpick was heroic: trying to save the situation, sacrificing itself, screaming, and when nobody helped, not the government, not the owners, not ... the building collapsed and all the damage was done.
But it's even more stupid than just that. As soon as this gets introduced, and some company makes a security fix. They of course, for GPL or AGPL software, have to release their fixes. This will then make them liable for any other security problems in that same software. After all, they'll be the last ones releasing that software after the government implemented this law.
So how will you even do this, without making software fixes effectively illegal? Achieving the exact opposite of what such a law tries to achieve ... But of course, you can't have this discussion with people just looking to keep "their" free stuff but trying to shift the rest of the costs.
The Debian project has completed a general-resolution vote, adopting a statement expressing concern about the Cyber Resilience Act (CRA) pending in the European Union.
Even if only "commercial activities" are in the scope of CRA, the Free Software community - and as a consequence, everybody - will lose a lot of small projects. CRA will force many small enterprises and most probably all self employed developers out of business because they simply cannot fulfill the requirements imposed by CRA. Debian and other Linux distributions depend on their work. If accepted as it is, CRA will undermine not only an established community but also a thriving market. CRA needs an exemption for small businesses and, at the very least, solo-entrepreneur.
I wouldn't mind receiving insurance coverage (and the background check required to support it) IF YOU PAID ME TO DO SO!
But we (mostly) don't even pay open source developers to write the code ... who is offering to pay them for this insurance?
Besides, this was a highly sophisticated actor. Someone willing to create several layers of loaders and invest long amounts of time into getting xz excluded from certain checks. Anyone with such sophisticated spy craft could have fooled the insurance companies also.
Expecting liability coverage for source code people publish for free on their own time has very strong implications on free speech, freedom of arts, and freedom of science. I don’t think this is possible in a liberal society.
On the other hand, you can already buy software, where the vendor takes some kind of liability: just buy Windows, AIX, or one of the commercial Linux offerings. Same for software libraries: there are commercial offerings that come with (limited) liability from the vendor. It’s even possible to create software to stronger standards. But outside of some very specific industries (aerospace, automotive, nuclear, defense, …) there doesn’t seem to be a market for that.
This is an easy way that will achieve the goal of completely killing free software, destroying the entire software industry in the process.
I contribute stuff for fun, for free. Now I also have to PAY to do that??? Plus anyone can just steal my identity… I have to show my ID every time I sleep at a hotel. Hundreds of people have a copy of my id and could use it to open an account in my name online…
Do you guys ever read what you write? Did you stop to think about it for more than 0.3 seconds?
I actually meant quite the opposite: that contribution should be paid. Yes, it would have to be ring-fenced so that society and the ecosystem would know who contributes what. That would also mean though that someone assumes liability for a piece of code; when you do that, you add value (economic not just source-code) and thus you should / have to be paid --by whom? the hundreds of commercial companies that use your code and whose liability you are reducing.
But every piece of software is legally held warrentless - no warranty is the heart of Microsoft, Oracle and the GPL licenses.
Yes I know the stories of “insurance made steam boilers safer”. And it’s true. But it also stopped innovation in the space before Charles Parsons came along and ignored the whole thing (military industrial aristocracy)
I think the answer sits somewhere in “have less stuff”.
We have millions of lines of code in all walks of life and Inswear we are orders of magnitude over engineered in almost all cases.
If you work for a large company try counting how many different ETL solutions exist, CSV uploaders, data lakes, warehouses and so on
Agreed and I dont believe "no warranty" can last that much longer, or in fact should. It was encouraged back in the day when all this computer stuff was new and either walled-off in unis or enterprises or in hobbyist's basements. But the real risk now is in the interconnections; the potential impact is order of magnitudes larger.
The closest metaphor is cars I think. And yes you can argue that innovation in cars has slowed down but also a 'minimum floor' of safety and efficiency forced by governments and insurers has made new entrants more likely. I.e. you shouldn't need to only trust Oracle, SAP with your business because then, erm, you'd have exactly the current situation in enterprise software...
>Agreed and I dont believe "no warranty" can last that much longer, or in fact should. It was encouraged back in the day when all this computer stuff was new and either walled-off in unis or enterprises or in hobbyist's basements. But the real risk now is in the interconnections; the potential impact is order of magnitudes larger.
Ok, I can blow your mind.
You can start your own software projects and offer them with warranty. And people can join you, if they want.
Why wouldn't people keep making open source, say "hey, no warranty!", but companies that use it in "load bearing contexts" have to assume liability for their choices, assuming someone enforces that.
Isn't that pretty much the way the world works now? What needs to be fixed?
> If companies insist on always taking the easy way out, they continue to introduce a significant amount of risk. Since it’s currently possible to make huge buckets of money, by writing crap, we’re not incentivizing high-Quality engineering.
For the past 10-15 years there has been a strong culture of never writing code when a library on GitHub or NPM already exists, which has, in large part, contributed to this. In many cases using existing battle tested code is the right thing, but it was taken to an extreme where the avoidance of pulling down a bunch of random open source packages with questionable stability and longevity was often maligned as not invented here syndrome.
Now many of those packages are unmaintained, the job hopping SWEs who added them to the codebase are long gone, and the chickens are coming home to roost. The pendulum will probably swing too far in the other direction, but thoughtful companies will find the right balance.
“…comes as-is, without warranties and without any commitment for future work. Complaints will get your feature request deprioritised, may get you banned, and will look silly to any potential employer googling your name”.
Also, let’s make it a meme to call out unreasonable behaviour: stop Jigar-Kumaring!
Perhaps the situation would improve if it were easier/more normalised to offer to pay the core developer to fix the bug that affects you. If that were the case, it would boil down to put up or shut up.
This wouldn't be entirely without downside though, as there could be a risk that the project ends up getting steered by whoever has the most money, which may be at odds with what the broader community gets from the project. That's difficult to avoid whenever Open Source developers get paid, unfortunately. If it were limited to bug-fixes I think the risk would be slim. I'm not sure if any projects have tried this.
I think the abuse is unfixable. Being exposed to many people just exposes you to many strange people. It is like how celebrities get paranoid and try to not be seen, since they are magnets for strange people that recognize them.
Being pseudo-anonymous filters out alot of credible threats though.
I'm honestly really worried about Andres. He thwarted a very expensive operation by an state actor.
Also, this backdoor was discovered by sheer chance. How many of those backdoors are still to be discovered? We should be checking all source code present in a standard Linux server right now, but all I can see is complacency because this single attempt was caught.
> thwarted a very expensive operation by a state actor
From the article:
..the fix for this was already in train before the XZ issue was highlighted, and long before the Github issue. The fix stopped the XZ backdoor into SSH, but hadn’t yet rolled out into a release of systemd.
I believe there’s a good chance the threat actor realised this, and began rapidly accelerated development and deployment, hence publicly filing bug reports to try to get Ubuntu and such to upgrade XZ, as it was about to spoil several years of work. It also appears this is when they started making mistakes.
> How many of those backdoors are still to be discovered?
Since keeping such backdoor hidden in plain sight is extremely hard and required tons of preparation and social engineering spanning multiple projects, the answer is probably a function of number of those already discovered. As we don't discover years-old similar backdoors every now and then and had discovered this one pretty quickly, this might very well be the very first one that came this far.
Also, what's "sheer chance" for an individual is "enough eyeballs" for a collectivity.
I think the fact that it happened pretty much by chance means he's not more of a threat to any state actor now than before. It's not like he's suddenly the anti-chinese-backdoor guy because of this. Or maybe he is, but more in a funny infosec hall of fame kinda way. It won't be him saving us next time.
> I'm honestly really worried about Andres. He thwarted a very expensive operation by an state actor.
I don't think Andres is in serious danger, unless he is a persistent threat to the bad actors. It's true that we owe him big time for discovering the backdoor. But it could have been someone else before him. And it may be someone else the next time. Too much depends on chance for anyone to justify targeting him. They risk blowing their cover by doing that just to satisfy their ego.
Also he’s a Microsoft employee in Seattle. Unless it was the NSA’s op, the U.S. government is unlikely to ignore a suspicious act in its territory and especially not the precedent that anyone else is allowed to mess with a key infrastructure provider’s employees.
he`s not launching a campaign against Unit 61398 and correlating everything they do to specific FOSS projects. he found a random bug and started asking a few questions. whacking him would accomplish nothing; any random nerd might have found this issue.
Code quality has been plunging for years while we've become more dependent upon code.
Devin AI working Upwork jobs blew minds, but it succeeded for a reason. Upwork and similar sites are plagued by low quality contractors who do little more than glue together code written by better engineers. It was never a hard formula to copy.
Outsourcing programming work to the lowest cost and quality to third-party libraries is leading to inevitable results.
Obviously, the next leap will be sophisticated supply chain attacks based upon poisoning AI.
I think it’s even more fundamental than culture, there are so many moving parts nowadays that the vast majority just aren’t smart enough to write high quality software within a normal 40 hour a week job.
e.g. The difficulty curve for writing apps went down a bit when iOS became popular, but nowadays iOS 17 is probably more complex than Snow leopard was in 2009 so there aren’t any low hanging fruit left for the median SWE.
Agreed.
> and face significant amounts of online abuse from users of the software.
Much as I’d like this to change, I suspect it will not. I’ve been doing open-source work since the mid-90s, and have suffered abuse that would curl your toes, but I still do it, for reasons that would be obscure, for many folks, hereabout.
I think the onus is on the users of the software. It’s currently way too easy to make “LEGO block” software, by splicing together components, written by others, that we do not understand (which, in itself, is not necessarily bad. We’ve been working this way for centuries). I remember reading a report that listed YAML as one of the top programming languages.
If companies insist on always taking the easy way out, they continue to introduce a significant amount of risk. Since it’s currently possible to make huge buckets of money, by writing crap, we’re not incentivizing high-Quality engineering.
I don’t think there’s any easy answer, as a lot of the most effective solutions are cultural, as opposed to technical. Yesterday, someone posted a part of an old speech, by Hyman Rickover[0], that speaks to this.
[0] https://news.ycombinator.com/item?id=39889072