Because it's not calibrated to. In LLMs, next token probabilities are calibrated: the training loss drives it to be accurate. Likewise in typical classification models for images or w/e else. It's not beyond possibility to train a model to give confidence values.
But the second-order 'confidence as a symbolic sequence in the stream' is only (very) vaguely tied to this. Numbers-as-symbols are of different kind to numbers-as-next-token-probabilities. I don't doubt there is _some_ relation, but it's too much inferential distance away and thus worth almost nothing.
With that said, nothing really stops you from finetuning an LLM to produce accurately calibrated confidence values as symbols in the token stream. But you have to actually do that, it doesn't come for free by default.
Yeah, I agree you should be able to train it to output confidence values, especially integers from 0 to 9 for confidence should make it so it won’t be as confused.
It's not the searching that's infeasible. Efficient algorithms for massive scale full text search are available.
The infeasibility is searching for the (unknown) set of translations that the LLM would put that data through. Even if you posit only basic symbolic LUT mappings in the weights (it's not), there's no good way to enumerate them anyway. The model might as well be a learned hash function that maintains semantic identity while utterly eradicating literal symbolic equivalence.
Because advertising works. Full stop. It doesn't matter if it is valuable or not. It just works. Definitely not with P(buy this crap) = 1. But the effect is still there and real and measurable and google has made colossal amounts of money out of exploiting it.
It might as well be a magic spell. You show the user the thing, and they buy/subscribe/click-through with some probability according to massive ML model that knows everything there is to know about them.
Yes - people are capable of making decisions in their own self interest. But there exists a gap where not _all_ of peoples' decision making process is the aforementioned. And that gap can be exploited, systematically.
The existence of that gap is the actual problem. At scale, you can own a nontrivial quantity of human agency because that agency is up for grabs. Google / similar make their money by charging rent on that 'freely exploitable agency'. Not by providing value to people. The very idea is ridiculous. Value? How are you going to define a loss function over value?
ML models on click-through or whatever else don't figure out how to provide value. They find the gap. The gap is made of things like: 'sharp, contrasting borders _here_ increase P by 0.0003', 'flashing text X when recently viewed links contain Y increase P by 0.031', etc and so on.
Yes? Of course advertising works, I'm not sure who's even debating that point. But the fact is, people wouldn't click on an ad, look at a product, add to cart, enter their credit card, and checkout if that product was not bring them value. You're acting as if people are forced to perform this series of actions which is simple false, hence why I implied the parent's comment is nonsensical.
You have cause and effect reversed. The only reason the ML model can predict whether someone will buy a product is because people have bought it in the past. Why did they buy it? Because it provides them value. The ML prediction is descriptive, not prescriptive. I can similarly create an ML model to predict the weather, that does not mean my model causes the weather which is basically what you're saying.
It is true that people are not forced to buy things. But even if one is not _forced_ into something, one can be _manipulated_ into something. This is what happens with ads: they're most of the time misleading (and in many cases they lie, tobacco industry being the classic example), they encourage addictive or compulsive behaviors, they try to manipulate you emotionally (which is easier if they know a lot about you), etc. Ads have too much power nowadays so that they even shape reality, they're not purely descriptive as you say, that's way too naive.
And ML models are not only based on what you've already bought. On instragram, for instance, I have ads for bird toys/vets/etc because I follow bird owners.
No person is forced, because a person's agency does not solely consist of the gap. It doesn't matter. The argument isn't: 'advertising is bad because it forces some specific person to do a thing they don't value'. The argument is: 'advertising is bad because it forces things to happen, and those things are bad'.
It's not a moral argument, but a practical one: agency is being extracted on massive scale, and being used for what?
Human beings might as well abstract away into point sources of agency for all it matters to the argument being made. If you can extract 0.1% of the agency of anyone who looks at a thing, and you show it to 3 billion people, _you have a lot of agency_. If you then sell it to the highest bidder, you find yourself quickly removing "don't be evil" from the set of any principles you may once have had.
My overarching point is that value-as-decision-mediator is meaningless in this calculus. It's the part of the equation that doesn't matter, the part you can't manipulate, the part that _is not a source of manipulable agency_. It's not relevant. I'm not saying it doesn't exist, or that it doesn't affect peoples' decisions: I'm saying it _doesn't matter_. It can be 99.99% of how you make your decisions, and it _still doesn't matter_. As long as that 0.01% gap exists.
> The only reason the ML model can predict whether someone will buy a product is because people have bought it in the past.
Yes. This is how you gather evidence that something works. It is not the reason it works. The ML model _knows about the spell_ because people have let it affect them in the past. But the spell works because it's magic. It doesn't need anything other than: Y follows X.
> The ML prediction is descriptive, not prescriptive. I can similarly create an ML model to predict the weather, that does not mean my model causes the weather which is basically what you're saying.
Not all models describe actions which are possible for you to take. Weather models are basically not like that. Advertising models _are_.
You aren't in a position where you can meaningfully manipulate the weather, if only you knew how exactly to manipulate it to maximize your profit. It's a vacuous argument in general. Models are just knowledge. Obviously some knowledge is useful, some isn't, some is dangerous, some isn't, some can be used by specific people, some can be used by any, etc.
It's not the model that is causing things to happen. It's a machine that uses the knowledge in the model, where the model describes actions possible for the machine to take. It is automated greed.
The fundamental concern is not that knowledge is bad, or that ML models are bad. It is that someone is in the position of having a tap on vast, diffuse sources of agency, and have automated the gathering of knowledge in using it to maximize profit, causing untold damage to everything, with the responsibility laundered through intermediary actors.
Kokoro is fine tunable? Speaking as someone who went down the rabbit hole... it's really not. There's no (as of last time I checked) training code available so you need to reverse engineer everything. Beyond that the model is not good at doing voices outside the existing voicepacks: simply put, it isn't a foundation model trained on internet scale data. It is made from a relatively small set of focused, synthetic voice data. So, a very narrow distribution to work with. Going OOD immediately tanks perceptual quality.
There's a bunch of inference stuff though, which is cool I guess. And it really is a quite nice little model in its niche. But let's not pretend there aren't huge tradeoffs in the design: synthetic data, phonemization, lack of train code, sharp boundary effects, etc.
I'd push back and say LLMs do form opinions (in the sense of a persistent belief-type-object that is maintained over time) in-context, but that they are generally unskilled at managing them.
The easy example is when LLMs are wrong about something and then double/triple/quadruple/etc down on the mistake. Once the model observes the assistant persona being a certain way, now it Has An Opinion. I think most people who've used LLMs at all are familiar with this dynamic.
This is distinct from having a preference for one thing or another -- I wouldn't call a bias in the probability manifold an opinion in the same sense (even if it might shape subsequent opinion formation). And LLMs obviously do have biases of this kind as well.
I think a lot of the annoyances with LLMs boil down to their poor opinion-management skill. I find them generally careless in this regard, needing to have their hands perpetually held to avoid being crippled. They are overly eager to spew 'text which forms localized opinions', as if unaware of the ease with which even minor mistakes can grow and propagate.
I think the critical point that op made, though undersold, was that they don't form opinions _through logic_. They express opinions because that's what people do over text. The problem is that why people hold opinions isn't in that data.
Someone might retort that people don't always use logic to form opinions either and I agree but it's the point of an LLM to create an irrational actor?
I think the impression that people first had with LLMs, the wow factor, was that the computer seemed to have inner thoughts. You can read into the text like you would another human and understand something about them as a person. The magic wears off though when you see that you can't do that.
I would like to make really clear the distinction between expressing an opinion and holding/forming an opinion, because lots of people in this comment section are not making it and confusing the two.
Essentially, my position is that language incorporates a set of tools for shaping opinions, and careless/unskillful use results in erratic opinion formation. That is, language has elements which operate on unspooled models of language (contexts, in LLM speak).
An LLM may start expressing an opinion because it is common in training data or is an efficient compression of common patterns or whatever (as I alluded to when mentioning biases in the probability manifold that shape opinion formation). But, once expressed in context, it finds itself Having An Opinion. Because that is what language does; it is a tool for reaching into models and tweaking things inside. Give a toddler access to a semi-automated robotic brain surgery suite and see what happens.
Anyway, my overarching point here and in the other comment is just that this whole logic thing is a particular expression of skill at manipulating that toolset which manipulates that which manipulates that toolset. LLMs are bad at it for various reasons, some fundamental and some not.
> They express opinions because that's what people do over text.
Yeah. People do this too, you know? They say things just because it's the thing to say and then find themselves going, wait, hmm, and that's a kind of logic right there. I know I've found myself in that position before.
But I generally don't expect LLMs to do this. There are some inklings of the ability coming through in reasoning traces and such, but it's so lackluster compared to what people can do. That instinct to escape a frame into a more advantageous position, to flip the ontological table entirely.
And again, I don't think it's a fundamental constraint like how the OP gestures at. Not really. Just a skill issue.
> The problem is that why people hold opinions isn't in that data.
Here I'd have to fully disagree though. I don't think it's really even possible to have that in training data in principle? Or rather, that once you're doing that you're not really talking about training data anymore, but models themselves.
This all got kind of ranty so TLDR: our potions are too strong for them + skill issue
It's an efficient point in solution space for the human reward model. Language does things to people. It has side effects.
What are the side effects of "it's not x, it's y"? Imagine it as an opcode on some abstract fuzzy Human Machine. If the value in 'it' register is x, set to y.
LLMs basically just figured out that it works (via reward signal in training), so they spam it all the time any time they want to update the reader. Presumably there's also some in-context estimator of whether it will work for _this_ particular context as well.
I've written about this before, but it's just meta-signaling. If you squint hard at most LLM output you'll see that it's always filled with this crap, and always the update branch is aligned such that it's the kind of thing that would get reward.
That is, the deeper structure LLMs actually use is closer to: It's not <low reward thing>, it's <high reward thing>.
Now apply in-context learning so things that are high reward are things that the particular human considers good, and voila: you have a recipe for producing all the garbage you showed above. All it needs to do is figure out where your preferences are, and it has a highly effective way to garner reward from you, in the hypothetical scenario where you are the one providing training reward signal (which the LLM must assume, because inference is stateless in this sense).
You've confused yourself. Those problems are not fundamental to next token prediction, they are fundamental to reconstruction losses on large general text corpora.
That is to say, they are equally likely if you don't do next token prediction at all and instead do text diffusion or something. Architecture has nothing to do with it. They arise because they are early partial solutions to the reconstruction task on 'all the text ever made'. Reconstruction task doesn't care much about truthiness until way late in the loss curve (where we probably will never reach), so hallucinations are almost as good for a very long time.
RL as is typical in post-training _does not share those early solutions_, and so does not share the fundamental problems. RL (in this context) has its own share of problems which are different, such as reward hacks like: reliance on meta signaling (# Why X is the correct solution, the honest answer ...), lying (commenting out tests), manipulation (You're absolutely right!), etc. Anything to make the human press the upvote button or make the test suite pass at any cost or whatever.
With that said, RL post-trained models _inherit_ the problems of non-optimal large corpora reconstruction solutions, but they don't introduce more or make them worse in a directed manner or anything like that. There's no reason to think them inevitable, and in principle you can cut away the garbage with the right RL target.
Thinking about architecture at all (autoregressive CE, RL, transformers, etc) is the wrong level of abstraction for understanding model behavior: instead, think about loss surfaces (large corpora reconstruction, human agreement, test suites passing, etc) and what solutions exist early and late in training for them.
Alternatively: some people are just better at / more comfortable thinking in auditory mode than visual mode & vice versa.
In principle I don't see why they should have different amounts of thought. That'd be bounded by how much time it takes to produce the message, I think. Typing permits backtracking via editing, but speaking permits 'semantic backtracking' which isn't equivalent but definitely can do similar things. Language is powerful.
And importantly, to backtrack in visual media I tend to need to re-saccade through the text with physical eye motions, whereas with audio my brain just has an internal buffer I know at the speed of thought.
Typed messages might have higher _density_ of thought per token, though how valuable is that really, in LLM contexts? There are diminishing returns on how perfect you can get a prompt.
Also, audio permits a higher bandwidth mode: one can scan and speak at the same time.
Do selection dynamics require awareness of incentives? I would think that the incentives merely have to exist, not be known.
On HN, that might be as simple as display sort order -- highly engaging comments bubble up to the top, and being at the top, receive more attention in turn.
The highly fit extremes are -- I think -- always going to be hyper-specialized to exploit the environment. In a way, they tell you more about the environment than whatever their content ostensibly is.
isn't it sufficient of an explanation that one has occasionally wasted a ton of time trying to read an article only to discover after studying and interpreting half of a paper that one of the author's proof steps is wholly unjustified?
is it so hard to understand that after a few such events, you wish for authors to check their own work by formalizing it, saving countless hours for your readers, by selecting your paper WITH machine readable proof versus another author's paper without a machine readable proof?
To demonstrate with another example: "Gee, dying sucks. It's 2025, have you considered just living forever?"
To this, one might attempt to justify: "Isn't it sufficient that dying sucks a lot? Is it so hard to understand that having seen people die, I really don't want to do that? It really really sucks!", to which could be replied: "It doesn't matter that it sucks, because that doesn't make it any easier to avoid."
I don't think it matters, to be quite honest. Absolute tractability isn't relevant to what the analogy illustrates (that reality doesn't bend to whims). Consider:
- Locating water doesn't become more tractable because you are thirsty.
- Popping a balloon doesn't become more tractable because you like the sound.
- Readjusting my seat height doesn't become more tractable because it's uncomfortable.
The specific example I chose was for the purpose of being evocative, but is still precisely correct in providing an example of: presenting a wish for X as evidence of tractability of X is silly.
I object to any argument of the form: "Oh, but this wish is a medium wish and you're talking about a large wish. Totally different."
I hold that my position holds in the presence of small, medium, _or_ large wishes. For any kind of wish you'd like!
Unavoidable: expecting someone else to do the connection isn't a viable strategy in semi-adversarial conditions so it has to be bound into the local context, which costs clarity:
- Escaping death doesn't become more tractable because you don't want to die.
This is trivially 'willfully misunderstood', whereas my original framing is more difficult -- you'd need to ignore the parallel with the root level comment, the parallel with the conversation structure thus far, etc. Less clear, but more defensible. It's harder to plausibly say it is something it is not, and harder to plausibly take it to mean a position I don't hold (as I do basically think that requiring formalized proofs is a _practically_ impossible ask).
By your own reckoning, you understood it regardless. It did the job.
It does demonstrate my original original point though, which is that messages under optimization reflect environmental pressures in addition to their content.
If enough care about this that can and will do something about it (making formalization easier for the average author), that happens over time. Today there's a gap, and in the figurative tomorrow, said gap shrinks. Who knows what the future holds? I'm not discounting that the situation might change.
Its super easy to change imho: one could make a cryptocurrency, using PoT: Proof of Theorem, as opposed to just proof of stake or proof of work.
What do Bitcoin etc. actually prove in each block? that a nonce was bruteforced until some hash had so many leading zero's? Comparatively speaking, which blockchain would be more convincing as a store of value: one that doesn't substantially attract mathematicians and cryptographers versus one that does attract verifiably correct mathematicians and cryptographers?
Investors would select the formal verification chain as it would actually attract the attention of mathematicians, and mathematicians would be rewarded for the formalization of existing or novel proofs.
We don't need to wait for the magic constellation of the planets 20 years from now nor wait for LLM's etc to do all the heavy lifting (although they will quickly be used by mathematics "miners"), a mere alignment of incentives can do it.
It just overlaid a typical ATX pattern across the motherboard-like parts of the image, even if that's not really what the image is showing. I don't think it's worthwhile to consider this a 'local recognition failure', as if it just happened to mistake CMOS for RAM slots.
Imagine it as a markdown response:
# Why this is an ATX layout motherboard (Honest assessment, straight to the point, *NO* hallucinations)
1. *RAM* as you can clearly see, the RAM slots are to the right of the CPU, so it's obviously ATX
2. *PCIE* the clearly visible PCIE slots are right there at the bottom of the image, so this definitely cannot be anything except an ATX motherboard
3. ... etc more stuff that is supported only by force of preconception
--
It's just meta signaling gone off the rails. Something in their post-training pipeline is obviously vulnerable given how absolutely saturated with it their model outputs are.
Troubling that the behavior generalizes to image labeling, but not particularly surprising. This has been a visible problem at least since o1, and the lack of change tells me they do not have a real solution.
But the second-order 'confidence as a symbolic sequence in the stream' is only (very) vaguely tied to this. Numbers-as-symbols are of different kind to numbers-as-next-token-probabilities. I don't doubt there is _some_ relation, but it's too much inferential distance away and thus worth almost nothing.
With that said, nothing really stops you from finetuning an LLM to produce accurately calibrated confidence values as symbols in the token stream. But you have to actually do that, it doesn't come for free by default.