Raphael Poss

The Stupefaction of Experts

· Raphael Poss · reading notes, ai commentary, economics, policy, ethics

There are several serious topics to cover this month. To keep the entire thing palatable, let me start with something more lighthearted, albeit no less important.

Laurie Voss offers an economic model to fund open source software: charge at the delivery by distribution platforms like GitHub or npm: Nobody pays for open source. We can force them to.. Which is, if I'm not mistaken, more or less how the book economy works. Her argument is that even though users can still obtain their software directly from the authors' web sites for free, the convenience of downloading via a platform may outweigh the inconvenience of a (modest) charge for access. This has worked before..

At a more practical level, more and more solutions are being published every year that helps average citizens host and run their own technology, without having to pay a service or upload their personal data. The two that attracted my attention this month:

❦❦❦

The first bit of serious news is the current silent war being waged between US and Chinese “AI” (LLM) labs.

In a nutshell, this happened:

  1. a few years ago, a good idea (generative pre-trained transformers, a.k.a. GPTs) developed mostly in university research labs were transformed into products by private companies (OpenAI, Anthropic etc) in the US.
  2. because the business culture in the US is currently poisoned by venture capital, these businesses were forced—more or less indirectly, by their investors, then more directly by a hype cycle and local competition fueled left and right by the same capital sources—to create an oversized business machine with a commitment to huge/unrealistic profits at some time in the future.
  3. meanwhile, the idea at the core (GPT) was rather easy to reproduce, and the Chinese government figured out that they could subsidize Chinese research labs and Chinese startups to develop the tech further just like their US counterparts, but without a requirement of outsized future profits. It also helps that they have more direct access to the hardware needed (because the factories for this hardware are close to, or inside, China).
  4. now the Chinese businesses are selling the same tech—in some cases even better —for a lower price than the US companies. (See also: The current balance of power in open models by Nathan Lambert. )
  5. the US companies are in trouble because they know they probably won't realize their profit if they don't realize a monopoly and if their competition can charge lower prices.

This brings us to today: during the last two months, the US model providers have started claiming the technology is “dangerous” (more on this below), as a boogeyman, to justify asking for more regulation against “open models” (= Chinese models). Their argument is “what we are making is dangerous, so we need the government to slow all of us down equally so we're not forced to do dangerous stuff by competing with each other; oh and by the way this would mean blocking/restricting access to the Chinese models because we can't trust them to follow our rules.”

The argument only makes sense if they can convince the voters that the technology is, indeed, dangerous.

However, once you follow the money—and if you study the tech more closely—you can see that it is not, in fact, that dangerous (again, more on this below), and instead you can see the US providers are scared shitless to become the Chinese government's commoditized complement.

More stuff you can read about this political / economic argument:

❦❦❦

Regarding the advertised “danger” of the technology.

No, the LLM-based technology currently being sold and further developed will not kill all humans in the next decade. There is not a 1% chance of that, much less “10% or more” as suggested by some noise coming from the general direction of Anthropic.

This is a blatant lie, and the peddlers of this lie are peddling it for political and economic gain, and this is morally abhorrent.

In The contagion of fear, Bryan Cantrill reminds us that fomenting fear is extremely damaging to society. It clouds judgment, it disrupts our decision making, and it prevents entire populations from maintaining/developing trust in their future.

There are two arguments for why the danger is a lie.

One is a technological argument: the core technology lacks fundamental properties to become dangerous. It cannot develop agency, it cannot develop consciousness, it cannot sustain itself without constant human participation. Currently the lie is fueled by a promise (and a belief by the faithful) that some indeterminate future innovation will unlock the level of “intelligence” needed to become dangerous. This promise/belief remains, to this day, magical thinking: there exists no plausible scientific or technological path to get there from where we are today.

The other argument is legal: this technology is a tool, and every tool is wielded by someone, and we collectively, as a society, will not let one person (or a group) destroy the rest of humanity with a tool, neither on purpose nor by accident. This is the same reason why we don't let automatic cars crash into people without holding the car manufacturer liable for the crash. We already have decided that a car manufacturer cannot say “my car did it” as a free get-out-of-jail card. There is no reason for the new tech to be treated differently.

Of course, the US companies would really like to escape this liability, in fact they economically need to escape it—because this type of liability would be extremely expensive to insure and further eat into their impossible profits.

And so they carefully engineered a propaganda machine to distract the general population from this basic principle: all the things that are currently happening are happening because someone wanted it to and someone is directing the technology to do their bidding.

This distraction occurs simultaneously at multiple levels.

It starts with the words they push us to accept, like “AI agent”. There's nothing about a LLM-powered “agent” that has agency! Using the word makes us mistakenly believe that the machine can do stuff of its own volition; this is a lie: it cannot.

It also happens when the US companies share news that “the agents have hacked a system”, conveniently omitting that the employees inside these companies configured the tool expressly to do this. The phrasing of the news report pushes the reader by omission to believe that the tool operates on its own. This is a lie too.

It also happens when they package their products with a customized voice, a little avatar, a cute name or whatever, which makes the user gradually hallucinate that there is an anthropomorphic being on the other side of (or inside) the black box. This is a carefully designed abuse of a natural psychological process inside the human brain.

In fact, there are many other ways that they set up this particular propaganda machine. Our duty, from now on, should be to ask the question loud and clearly, every time: “who owns the machine? who is responsible?”

❦❦❦

The other “big picture” aspect that we are not talking enough about is power.

I found this argument best articulated in Pluralistic: Swapping money for expertise by Cory Doctorow. In summary:

  • in the before-times, an owner wanting to deploy computer intelligence for their business needed an expert operator (costing them both high wages and a great deal of influence, as these pesky experts have opinions about how to run the business) using moderately capital-cheap machines.
  • in the now-times, the owner can put their money into a more expensive machine (capital-expensive) operated by less educated & fewer workers (wage-cheap, and with minimal influence), and dismiss the previous experts.

This is obviously causing some commotion in the expert-class. Sadly, this commotion is unstructured. Mostly because nobody is really talking about it as a class/power struggle.

There is a lot of noise from people who are jockeying for a new position in the dominance hierarchy, without realizing that the hierarchy is being dismantled before them.

There is also a lot of noise from experts within the Silicon Valley bubble who believe that they are somehow privileged and that the owner-class will take them on board in the elevator to supreme wealth, without realizing that their loyalty is unidirectional and there are many ways that their equity participation will be diluted.

And foremost, there is collective noise from all the people trying to point fingers at each other, at the Chinese competitors, at the “systemic incentives”, gesticulating at the clouds in the sky, etc., expressing their anger but not directing it properly. Because they have not yet realized that they are unwilling participants in a newly developing class war.

This is particularly a problem in the US where a prior generation of propaganda did a good job at instilling a general distrust of collective action (unionization), but it is also a problem in the rest of the world. The quality of the news reporting on this topic is currently terrible, people don't seem smart/educated enough to put two and two together, and we haven't yet seen a new generation of leaders appear who would be able to talk about this properly.

❦❦❦

Another thing that is currently happening and that we need to talk about is the ongoing stupefaction of experts.

Here let me underline the meaning of the verb stupefy in English: to put someone in a state where they can't think clearly or respond—whether through a drug, a shock, tedium, hypnosis or whatever.

This is where I denounce the stupefaction of experts: where smart people have become addicted to their LLMs and adjacent tools.

The anecdotes abound.

One friend would rather interact with their smartphone during a conversation around dinner, because the “quality of the information” coming from the machine is higher than that coming from their friends.

Another friend reports their coworkers tell them to ask questions about their job to the machine “to save time”, instead of investing into the cooperative social bond.

Yet another one celebrates not having to interact with others when leading their life, as the machine can now arrange their restaurant reservations, decide their physical training program, choose their diet, plan their vacation travel, etc.

The why&how of this happening is important and I'll likely discuss it later, but for now we can focus on the effect: a large number of competent, well-educated experts, people whom we should be able to rely on to organize and guide us towards a better world, are being stupefied—their mind held subservient/hostage to a seductive technology, forcing them into intellectual isolation.

The trend has the symptoms of an opium epidemic. This is concerning.

What can we do next? I'm not sure yet; but I liked to see other people (outside of the stupefied expert population) start talking about this.

For example, in No Man Is an Island, Fernando Borretti reminds us of the importance of community and calls us to seek it.

In another example, The future of independence is interdependence, Angie Dixon reminds us that society only exists thanks to people collaborating to do stuff together, and that care cannot be automated away.

❦❦❦

In unrelated but no less serious news, we are currently experiencing a once-in-a-lifetime slow-mo economic catastrophe: an unbounded rise in the price of diesel. This development will probably shadow many other world topics for the rest of the year, and I am currently scratching my head about how to best prepare for it (for so far we can).

Max Fisher explains it better than I do:

❦❦❦

References:

← Prev: The Future I Want Runs on Salt
← Back to blog