Old operating systems like VMS, MULTICS, OS/400, etc. aimed for this; they were large and contained support for a lot of structure. Problem is, they always evolved to be too complex, and/or the available complexity was not what turned out to be needed, but different complexities had to be built on top of the old, unneeded, complexities.
Along comes Unix and dispenses with all of that. Files? Bytes only. Devices? Accessible as files in the file system. Etc. This became popular for a reason.
I came here to write this. VM/CMS was designed for processing structured data. Application UIs were designed around forms to be displayed on 3270 terminals and data was structured around what could be input on a single punch card. It was great as long as this fit your model.
What UNIX gave the world was maximum flexibility: an os that only really cared about streams and got out of your way.
And in my eyes this is the only right way, because it allows to build structured services on top. If on the other hand the structure is already in the underlying system, it's incredibly hard to build something useful on top.
Similarly, think how useful memcpy() is: Because it can be applied anywhere.
The important question here is, why? Was it because of those features? Or something else?
I don't know, but one major factor in OS adoption is network effects. "The rich get richer", and it's very hard to introduce a new and different system, no matter how good it is.
Unlike Unix, which was basically free software and therefore extendable and available for companies to build their own versions of, Plan 9 was proprietary from Bell Labs.
Having cruft in your top layer is fine and healthy; a sign of a living, evolving system that is still learning how best to do its job and hasn’t prematurely ossified.
The problems occur when a still-crufty layer becomes set in stone, ensuring its base defects go on to pollute everything built on top. That may happen either because its designers lack perspective and cannot accept their precious design is flawed; or because everyone else starts building on top before it is ready for prime-time, then reject all attempts to straighten it out because that would require them to rewrite as well.
Worth bearing in mind here that files and file systems are themselves a kludge, a 1950s workaround for not having enough primary storage to keep all programs and data live at all times.† (Also, primary storage traditionally tends to be volatile; not good when the power goes out.) I point this out because institutionalizing kludges as The Way Things Should Be Done is an awfully easy mistake to make, and in a long-lived systems like OSes have serious architectural sequelae.
..
What’s interesting about the Unix File System is that it’s a general abstraction: a hierarchical namespace that can be mapped to a wide range of resources; not just “files” in secondary storage but also IO devices, IPC communication points, etc. And that’s all it did: mount resources at locations and define a common API for reading/writing/inspecting all resources, without getting bogged down on the internal details of each resource. Nice high cohesion, low coupling design.
Plan 9 made much fuller use of the namespace idea than Unix did, but the core concept was there for the start and it is excellent… except for one MASSIVE mistake: individual resources are neither typed nor tagged.
Without formal type information there is no way for a client to determine the type of data that a given resource represents. Either there is a common informal agreement that a resource mounted at location X is of type Y (e.g. /dev/*), OR there is an informal resource naming convention (typically DOS-style name extensions), OR the client has to guess (e.g. by sniffing the first few bytes for “tells” or by assuming ASCII).
Formally tagging each resource with, say, a MIME type (as in BeFS, or properly implemented HTTP) would’ve made all the difference. THAT is K&R’s million-dollar mistake mistake, because without that key metadata it is impossible to ensure formal correctness(!) or implement intelligent helpers for data interchange (e.g. automatic coercions).
Arguments over the pros and cons of alternative namespacing/association arrangements (e.g. relational vs hierarchical) are all secondary to that one fundamental cockup.
..
Unix became popular not because it was Good, but because it was Just Good Enough to entertain the cowboys who built and used it, who enjoy that sort of pants-down edge-of-the-seat living. And because a lot of them were in education, they spread it; and so made cowboy coders the status quo for much of this profession and culture. And while I admire and largely approve of their Just Do It attitude, I abhor their frightening disregard for safety and accountability.
And while I’m heartened to see some signs of finally growing up (e.g. Rust), there is a LOT of legacy damage already out there still to be undone. And retrofitting fixes and patches to endemic, flawed systems like Unix and C is and will be infinitely more pain and work than if they’d just been built with a little more thought and care in the first place.
--
† If/When memristors finally get off the ground, the primary vs secondary storage split can go away again and we finally get back to single flat storage space, eliminating lots of of complex bureaucracy and ritual. And when it does, there’ll still the need for some sort of abstract namespace for locating and exchanging data.
> a 1950s workaround for not having enough primary storage to keep all programs and data live at all times
Even today you don't have enough primary storage to keep your data. And even then it would require a structure when data outlives the process / application.
Most times there are no best solution, but only tradeoffs. Anyone who has done a bit of systems work knows this. And Hierarchical file system was a Ok-ish trade-off to make. Perfect is enemy of good.
> MASSIVE mistake: individual resources are neither typed nor tagged.
It comes with its own set of tradeoffs. I am a huge proponent of static typing when it comes to PLs. But in a system where multiple actors operate on shared resources, it is easy to get illusioned into a false sense of correctness. Also it imposes some extra complexity in the programming model. I am no experienced systems engineer. But someone here can address it better.
> .... entertain the cowboys who built and used it ....
You are going beyond HN standards to justify your anger against a particular methodology or people that embrace it in programming.
The universally accepted point is that Unix succeeded due to political factors (low cost and easy modification compared to proprietary counterparts), simplicity of the API, and being arguably better than others despite lacking some features people love to lament these days. But in many cases, that simplicity is a desirable thing to have. It is nice to objectively point out faults in systems. But what you did is totally dismissing some people's contributions.
It is easy to see some hyped thing and think that's the Next Big Thing(TM) after reading two fanboys preaching on Reddit, while being totally ignorant of tradeoffs.
> Unix became popular not because it was Good, but because it was Just Good Enough to entertain the cowboys who built and used it ... and so made cowboy coders the status quo for much of this profession and culture
I disagree very strongly with these insults directed at programmers from 50 years ago because now, in retrospect, half a century later what they did doesn't live up to some flawless system written in Rust that exists only in one's imagination.
Doesn't is seem a little bit like calling Thomas Edison a cowboy that made the terrible mistake of giving us electric lighting through filament bulbs when LED lights would have been so much better.
In these early days of computer science I read virtually every important published article on programming languages and operating systems, the field was still that small. MIT didn't even think it warranted a separate department, it was just a subsidiary branch of EE like say communications. Researchers like Edsger Dijkstra, Tony Hoare, Niklaus Wirth, Per Brinch Hansen, Leslie Lamport, David Gries, Donald Knuth, Barbara Liskov, and David Parnas were all trying to figure out how to structure an operating system, how to verify that a program was correct, how to solve basic problems of distributed systems, and how to design better programming languages. Practitioners working on operating systems would have been familiar with almost everything written by these giants.
It's easy to insult C, I myself wouldn't choose it for work today. But in 1989, 20 years after the birth of Unix, I did choose it for my company's development of system software--it still made sense. And back in the 1960's what alternatives were there? Fortran? PL/1? Pascal? Lisp? We were still programming on keypunch machines and relational databases hadn't been invented. The real competition back then for system programming was assembly language.
> Formally tagging each resource with, say, a MIME type (as in BeFS, or properly implemented HTTP) would’ve made all the difference.
That gives you a "global naming of things" problem, which is surprisingly hard. Who controls the namespace? Who gets to define new identifiers? Do they end up as intellectual property barriers where company A can't write software to work with files of company B?
> without that key metadata it is impossible to ensure formal correctness(!)
That seems irrelevant - even with the metadata you have to allow for the possibility of a malformed payload or simple metadata mismatch. I don't believe this alone would prevent people from sneaking attack payloads through images or PDFs, for example.
> THAT is K&R’s million-dollar mistake mistake
K&R wrote UNIX before MIME. Not only that, but before the internet, JPEG, PDF, and indeed almost all the file types defined in MIME except plain text.
Refusing to choose also prevented UNIX from being locked into choices that later turned out to be inconvenient, like Windows deciding to standardise on UCS-2 too early rather than wait for everyone to converge on UTF-8.
Even the divergent choice of path separators and line endings has turned out to be a mess.
> cowboys
The "cowboy" system is the one that beat the others, many of which never launched (WinFS, competing hypertext protocols) or were commercially invisible (BeOS, Plan9 etc).
Both Windows and MacOS have alternate file streams which can be used for metadata, but very rarely are.
Memristors aren't going to save you either. Physical space ultimately determines response time. You can only fit so much storage inside a small light cone. We're going to end up with five or six different layers at different distances from the CPU, plus two or three more out over the network, getting cheaper and slower like Glacier.
We probably are going to move to something more content-addressable, in the manner of git blobs or IPFS, and probably a lot closer to immutable or write-once semantics because consistency is such a pain otherwise. It would be interesting to see a device offering S3-style blob interface plus content-addressable search ... over the PCIe interface.
Oh, and there's a whole other paper to be written on how access control has evolved from "how can we protect users from each other, but all the software on the system is trusted" to "everything is single-user now, but we need to protect applications from each other".
> That gives you a "global naming of things" problem, which is surprisingly hard. Who controls the namespace? Who gets to define new identifiers? Do they end up as intellectual property barriers where company A can't write software to work with files of company B?
Well, you could go the sqlite route: if you want the DB to be globally accessible for everyone else, pass an absolute path; if you want it to be anonymous and for your temporary usage only, pass ":memory:" as the DB path.
Additionally, UUIDs / ULIDs / what-have-you can be extended but IMO just make 512-bit global identifiers the norm and say "screw it" for the next 1_000 - 100_000 years. (While making a standard library to work with those IDs baked in every popular OS's kernel).
Sure, it might get complicated. Still, let it be one complicated thing on one specific topic. Still a better future status quo compared to what we have today.
---
I mean, nobody here seems to claim that changing things will be instantly easier and stuff will automagically fall in place in a year. No reasonable person claims that.
But freeing us from the "everything is strings on shell" and the "zero conventions" conundrums themselves will probably solve like 50% of all technical problems in programming and system/network administration.
(While we're at it, we should redefine what the hell a "shell" even is.)
Along comes Unix and dispenses with all of that. Files? Bytes only. Devices? Accessible as files in the file system. Etc. This became popular for a reason.