The month was so busy that I barely had time to read anything. The only bit of AI news that I felt was noteworthy was this article:
Evaluating Large Language Models in Scientific Discovery by Zhangde Song et al, in other words an astounding bunch of scientists across major research labs in the USA, Canada, Switzerland, China and the UK.
In a nutshell, the authors evaluated LLMs across the full loop of scientific discovery: hypothesis, experiment, observation, revision and back to hypothesis. They also evaluated them on tasks spanning multiple fields including biology, physics and chemistry.
What they found was… underwhelming. LLMs are good at proposing hypotheses, but disappointingly brittle at everything else. They overfit to patterns, they struggle to abandon bad hypotheses even in the face of contradictory evidence, they confuse correlation and causation, they hallucinate explanations when experiments fail, and they optimize for plausibility instead of truth.
More worryingly, high benchmark scores do not correlate with high scientific ability. The big, expensive models that were optimized to top reasoning tests can't seem to run iterative experiments and update hypotheses.
The main takeaway: scientific intelligence is not language intelligence.
There is still a huge gap.
❦❦❦
Another lesson that gave me serious pause, and which I now find very important to share, is this video:
The author is a Russian person living in Estonia. He teaches us how modern propaganda really works. According to him, propaganda is not based on blatantly false words anymore, said very loudly, with censorship of contrary opinions. Instead, it's using a cacophony of contradictory opinions to disable our interest in the truth, by overwhelming our attention. That feeling that you'd rather switch off the news on the TV/radio and instead play video games? That is what modern propaganda wants you to do.
I felt cold chills in my back when I understood what he was explaining. Back in March, I already started to feel that something was amiss, but I hadn't envisioned back then that this could be the effects of active propaganda.
My current project to help folk claim their attention back feels even more important in this new light.