> One possibility is that AGI remains a tool, without a mind or direction other than what we give it
"Solve for the equilibrium" is the name of the game. "AGI remain a tool" is nice, until you notice that some people will use it lightly (solve some day-to-day problems I encounter when running my business, but I still do most of the work and decisions), some people will use it less lightly (I give you complete access to my mail box and my bank account — now run my business for me), and the first one will get absolutely crushed by the second in any market that is remotely close to free.
If you can ensure that "AGIs remains a tool", you just get to the same result, "AGI controls the future", with an additional step of nominally-in-charge humans rubber-stamping decisions made by AGIs.
> I sometimes feel that the doomers lack imagination.
It’s really the opposite.
I have asked a lot of people, "doomers" or not, "what is the positive vision of a future where there are AGIs around that dominate human in all cognitive tasks ?". All the answers feel superficially good and correct, until you actually use your imagination to poke and probe at it. It always fall apart. The only exception is those accepting : "humans not in control of the future is good actually, I don’t mind having infinite entertainment and zero responsibility". I have nothing to reply to that except : I personally abhor that picture.
Did you really think I have not enough imagination to generate by myself the hypothesis "what if AGIs stay tools ?". Of course I have. It’s just that I have used my imagination to dig deeper at that image. And the result is still not the pretty happy story it initially looks like.
Once you generate a dozen of such hypothesis, dig, and find the same outcome, you start to get the feeling that it is not exactly an accident.
It is not a lack of imagination. The problem is fundamentally extremely hard, possibly impossibly hard.
> The world is chaotic. We have no way to predict that this is what it will end up like.
I strongly disagree with that. Chaotic behavior on the micro-scale is not incompatible with predictability on the macro-scale. Almost all science is like that actually. Even tho you can’t predict when and in which individuals a particular mutation will arise, and the exact life path of those individuals, you can still have laws relating relative fitness and fixation rates, and the prediction that eventually, a beneficial mutation will fixate in a population.
Everything you know works like that. Electromagnetism does not require knowledge of individual photon/electron interactions to describe Maxwell equations. Thermodynamics do not require detail knowledge of molecular dynamics to relate quantities like pressure and temperature.
Today we summarize the native/settlers conflicts by "Eventually the Settlers displaced the Natives from their lands". And while it leaves some nuances out, it is not a bad summary. If you ask a human living this period, he will say what you say "world is chaotic, who can predict what the outcome will be ?". Actually the overall outcome was pretty obvious given the significant technological gap. Details may be surprising ; the big picture isn’t.
Edit : I focus too much on AGI on this message, not enough on humans. The problem is at the end mostly a human problem. Accepting diversity — true diversity, "I abhor your vision for a future, but I agree you should have a place to realize it with other like-minded people, as long as you don’t interfere with other people realizing their vision of the future" is the cure. If humans can accept genuine diversity, I believe there are pretty straightforward solutions that are robust to at least my level of poking and probing. I do not expect humans to be able to accept genuine diversity tho, and the more I reach out to other humans, the more I despair on this particular axis. It’s very easy to make a human say "Yes Diversity is Good" in the abstract. Then you start to paint some concrete pictures, and the mask fall off: "actually the diversity I like means accepting some people slightly, superficially different from me".
"Solve for the equilibrium" is the name of the game. "AGI remain a tool" is nice, until you notice that some people will use it lightly (solve some day-to-day problems I encounter when running my business, but I still do most of the work and decisions), some people will use it less lightly (I give you complete access to my mail box and my bank account — now run my business for me), and the first one will get absolutely crushed by the second in any market that is remotely close to free.
If you can ensure that "AGIs remains a tool", you just get to the same result, "AGI controls the future", with an additional step of nominally-in-charge humans rubber-stamping decisions made by AGIs.
> I sometimes feel that the doomers lack imagination.
It’s really the opposite.
I have asked a lot of people, "doomers" or not, "what is the positive vision of a future where there are AGIs around that dominate human in all cognitive tasks ?". All the answers feel superficially good and correct, until you actually use your imagination to poke and probe at it. It always fall apart. The only exception is those accepting : "humans not in control of the future is good actually, I don’t mind having infinite entertainment and zero responsibility". I have nothing to reply to that except : I personally abhor that picture.
Did you really think I have not enough imagination to generate by myself the hypothesis "what if AGIs stay tools ?". Of course I have. It’s just that I have used my imagination to dig deeper at that image. And the result is still not the pretty happy story it initially looks like.
Once you generate a dozen of such hypothesis, dig, and find the same outcome, you start to get the feeling that it is not exactly an accident.
It is not a lack of imagination. The problem is fundamentally extremely hard, possibly impossibly hard.
> The world is chaotic. We have no way to predict that this is what it will end up like.
I strongly disagree with that. Chaotic behavior on the micro-scale is not incompatible with predictability on the macro-scale. Almost all science is like that actually. Even tho you can’t predict when and in which individuals a particular mutation will arise, and the exact life path of those individuals, you can still have laws relating relative fitness and fixation rates, and the prediction that eventually, a beneficial mutation will fixate in a population.
Everything you know works like that. Electromagnetism does not require knowledge of individual photon/electron interactions to describe Maxwell equations. Thermodynamics do not require detail knowledge of molecular dynamics to relate quantities like pressure and temperature.
Today we summarize the native/settlers conflicts by "Eventually the Settlers displaced the Natives from their lands". And while it leaves some nuances out, it is not a bad summary. If you ask a human living this period, he will say what you say "world is chaotic, who can predict what the outcome will be ?". Actually the overall outcome was pretty obvious given the significant technological gap. Details may be surprising ; the big picture isn’t.
Edit : I focus too much on AGI on this message, not enough on humans. The problem is at the end mostly a human problem. Accepting diversity — true diversity, "I abhor your vision for a future, but I agree you should have a place to realize it with other like-minded people, as long as you don’t interfere with other people realizing their vision of the future" is the cure. If humans can accept genuine diversity, I believe there are pretty straightforward solutions that are robust to at least my level of poking and probing. I do not expect humans to be able to accept genuine diversity tho, and the more I reach out to other humans, the more I despair on this particular axis. It’s very easy to make a human say "Yes Diversity is Good" in the abstract. Then you start to paint some concrete pictures, and the mask fall off: "actually the diversity I like means accepting some people slightly, superficially different from me".