Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

It works fine for rectangular structures, I mean you can write a JSON document like

  { "a": 1, "b": 2, "c": 3}
you do pay a performance cost but remember the thing can have a compiler in it that can figure out that the data that goes over a certain line has a fixed structure and treated accordingly, e.g. the above could be just

  01 02 03
The trouble w/ the relational model is that you often have to calculate things that involve joins (say you want to know the average selling point of items from the garden supplies department were bought by people in the rewards program) and you fundamentally can't do that inside one box, rather you have to build a great big network of boxes and lines that is difficult to understand and maintain. What is 8 boxes with joins could be done in one box if the data was shaped appropriately. You open up a whole world of possible software reuse (cut and paste one box, not a huge network of boxes and lines)

The demo I did processed XML documents from

https://www.gleif.org/en/lei-data/access-and-use-lei-data

which was almost like a flat CSV listing information about business registrations and then merged it with some other flat files and a "knowledge graph" to make a set of specialized databases that made a web application for browsing registrations.

A key concept here is "structural stability", yes 90% of the data comes in rectangular forms. With graphs you can deal with the other 10% with about 10% more effort but if you are in the rectangular straightjacket it is more like 10x more effort not 10% more. Pretty quickly most of the headspace in a classical data analysis pipelines involves the weird, irregular and hard to do stuff even if it's not most of it.



The relational approach requires flattening JSON or XML into a table. Which is far from ideal, I agree.

What you are suggesting sounds more general purpose, but also significantly more complex. A table of data that passes through transforms is quite easy to explain to non-techies. I don't fancy explaining your approach to a non-techie.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: