9 September 2026
Jacob Coxon, AI researcher.
- 1 -
Bill
Joy was a fluent reader by the age of three.
Eventually,
he co-founded a high-tech company in Silicon Valley and contributed
significantly to computer science.
In
April 2000, rather surprisingly, he published in WIRED Magazine a controversial
article, which alienated numerous scientists.
The
stirring article questioned cherished values and threatened broad self-interests.
The
title was no less unusual:
“Why
the Future Doesn't Need Us.”
Indeed,
Joy composed a lengthy thesis that outlined the “Pandora's boxes” of genetics,
robotics, and nanotechnologies.
He
was deeply worried about research programs, typically very promising
commercially, which endangered humanity's survival like weapons of suicide.
Implicitly
and explicitly, he blamed leading scientists for downplaying the dangers of
their research programs, or (no less alarming) taking appalling risks of which
they were well aware.
The
question he raised was plainly dramatic:
"Can
we doubt that knowledge has become a weapon we wield against ourselves?"
Practically,
Joy proposed to restrict the search for knowledge, and relinquish the most
hazardous types of breakthrough science.
But
if he hoped to shake the status quo, he proved to be rather naive.
The
sensation he created was rather short-lived.
He
did not inspire a notable social issue nor an official constraint on the
customary liberty of science.
The
firm tradition of government's lavish support for science combined with the public's
admiration of science hardly changed, if at all.
No
meaningful change was noted in the fundamental presuppositions and aspirations
of scientists.
In
short, scientists carried on in the same old ways, which was not a big
surprise.
Ground
breaking research programs had long been business as usual, at times rewarded
with gold medals.
- 2 -
Wild
rabbits inflicted heavy damage on Australian agriculture for quite a long time.
In the 1950s, professor Frank Fenner played a key role
in an official project, which exposed the rabbits to a lethal disease and
drastically reduced their numbers.
Later on, Fenner highly contributed to the worldwide
eradication of a devastating human disease, smallpox, officially announced in
1980.
Eventually, though, his broad concern with global
problems culminated in despair.
Like others, he noticed the huge quantities of modern
machines that pollute the air and disrupt the climate. But his calculations
showed a chilling consequence – or maybe better said a “scorching consequence”
– which he disclosed publicly shortly before his death.
Cheryl Jones covered a unique interview with Fenner in
“The Australian” on June 16, 2010. That day, many readers were surprised to
learn that the admired scientist supported the most radical claims of the most
pessimistic camps:
Global warming will escalate, mitigation efforts will
fail, and humanity will eventually face annihilation.
However, though the interview seemed quite
sensational, not every reader was actually surprised.
Some might have long followed the periodical Doomsday
Clock announcements.
Some might have carefully read a well-known concise book
by Martin Rees, who studied the global dangers of contemporary technologies and
gave a sobering assessment:
The risk of apocalyptic catastrophe in the 21st
century is at least 50%.
Rees, former president of the Royal Society, did not
answer a pretty simple question:
Will modern civilization survive or not?
But based on Rees, at best, the odds are even, like
the odds of a flipped coin landing on heads or tails.
- 3 -
History teachers might offhand note that starting in the 18th
century, consecutive scientific breakthroughs transformed human life across the
board.
No doubt, before that time, no one ever heard of a train
locomotive, antibiotic tablet, television, computer network, jet plane, phone, vaccine
shot, or electrical appliance.
Also before that time, no one ever heard of engine pollution,
cyber terrorism, grey goo accident, uncontrolled artificial intelligence, engineered
pandemic, or nuclear warfare.
At this moment, it is pretty hard to explain the precarious state
of affairs:
Human society confronts scores of existential dangers it has
itself created one by one.
The risk of human self-extinction, as staggering as it is
currently quantified, may still rise further.
Moreover, a random accident may trigger the end of humanity
even next week, as simple as that.
How could the most intelligent species so thoroughly jeopardize
its sheer existence?
How could it happen?
Here is a tentative list of partial answers:
* For millennia, scientists simply knew nothing about the
potential dangers of their regular agendas.
* For ages, scientists sidestepped systematic philosophical
warnings about the uncertainty and instability of human knowledge.
* No scientist ever risk-assessed the scientific enterprise
in toto.
* Outcomes of scientific development commonly looked
advantageous in the short range.
* Scientists misled by vast self-interests backed by false presuppositions
further misled government officials, military brass, U.N. delegates, entrepreneurs,
lawmakers, media reporters, and intellectuals.
* Inter-state hostilities, huge financial stakes, and legal complexities
routinely disrupted adequate control of scientific development.
* Defective scientific logic obscured by opaque scientific
jargon routinely disrupted adequate control of scientific development.
* Small, powerful elites covertly forced billions of humans not
only to support but also to celebrate patently foolish gambling.
- 4 -
The myth is pretty well known.
Following a futile siege of Troy, the Greek
army built a huge hollow sculpture of a wooden horse, hid some troops inside,
moved it close to the immense walls of the defiant city, and feigned a retreat.
Not long after, the Trojan leaders swallowed
the bait. The wooden horse was brought in, and then the concealed troops
sneaked out and opened the main gates to the whole Greek army, which swiftly
devastated Troy.
Briefly given in the Odyssey, this story has
been told and retold countless times traditionally in verse accompanied by
music.
For some, though, it was merely a peculiar
piece of dubious history.
Others were prompted to examine specific
issues of ingenuity and stupidity.
Based on Barbara Tuchman, the fall of Troy
was a prominent case of elite debacle, covered in her 1984 historical work:
The March of Folly: From Troy to Vietnam.
This book dealt with political policies and decisions
in many different circumstances and was met with some professional criticism.
Still, it drew readers’ attention to curious
questions:
Could some types of people tend to take
unwarranted risks?
Could excessive power breed not only straight
corruption – which is widely maintained – but also defective thinking?
Could foolishness strike regardless of IQ
level?
Such questions seem quite intriguing these days, when multiple global threats might lead to the self-extinction of the entire human society.
Victims
of Groupthink.
Janis
studied defective logic shaped by social factors in certain over-confident and
inflexible groups. He called it “group thinking,” or more contemptuously,
“groupthink.”
Could
“groupthink,” then, characterize some types of current social elites?
Given
the up-to-date Doomsday Clock setting, this question seems to merit some serious
attention, to say the least.
Interestingly,
Janis was influenced by George Orwell, who coined infamous terms like
“blackwhite” and “doublethink,” pointing to the twisted thinking of an
unbridled elite.
As if
to broaden the issue, Gorgias of Sicily (discussing the Trojan War) suggested a
linguistic insight on the basic development of twisted thinking, faulty
thinking, absurd thinking, and similar achievements of the human mind:
Words
are like drugs.
According to Nietzsche, when one notices the
sheer conviction upon which this or that philosophy is based, one can only say:
“The ass arrived, beautiful and most brave.”
The Latin words he actually used, citing a line
of a famous Christian song, seem somewhat more spicy:
“Adventavit
asinus, pulcher et fortissimus.”
Just for
the record, Nietzsche fully supported his scornful dismissal of celebrated
minds with an array of astute arguments.
Perhaps
the most comprehensive review of stupidity-at-the-top was given by the
Renaissance humanist Erasmus Desiderius, who used his subversive wit to portray
the world he knew, and maybe just as well – given some persistent evidence –
the contemporary world all of us know.
Based
on his 16th-century classic, The Praise of Folly, planet Earth is ruled by no
other than the goddess of foolishness.
All
humans worship this goddess, all are happy, and all are fools. But the question
remains: who are the greatest fools of all?
As to
this particular issue, Erasmus Desiderius carefully lists great philosophers,
famous writers, respectable educators, commerce magnates, and theologians
officially licensed to eradicate any sign of heresy. In short, the greatest
fools of all are precisely those considered the wisest of all.
Whether
this very model fits the contemporary world is, of course, up to anyone to
decide.
The catalog of the recognized agents of folly,
so to speak, is substantially longer. Prominent samples, aside from straight
fools, are donkeys, simpletons, dolts, idiots, imbeciles, buffoons, ninnies,
and dummies, not to mention goons, loons, jackasses, noodles, schmucks, simps, palookas,
and poops.
To be sure, this is just one small hint on
the vast topic of folly.
Historically, the defective functioning of
the human brain – the brain of the most intelligent species – so the story goes
– has been a spectacle observed in traditional folklore dating back millennia.
It has further been discussed by
philosophers, historians, novelists, theologians, poets, logicians, comedians,
psychologists, and military theorists, not to mention kindergarten nurses.
Whether it is a problem to be solved, a
weakness to be exploited, a vice to be ridiculed, or just a plague, depends on
the point of view.
Whether some significant progress has been
achieved in this respect is hard to tell.
If someone is eager to write a concise treatise
on the subject, I would recommend a title like this:
“The March of Folly from the Tree of
Knowledge to the Tsar Bomba – the Most Powerful H-Bomb Ever Detonated.”
The monstrous Tsar Bomba, by the way, was chiefly
designed by Andrei Sakharov, a proud Nobel Prize winner, honored for his
unparalleled contribution to the “peace” of humankind.
Starting in 1901, the Nobel Prize was awarded to foremost
scientists in view of their accomplishments for the great benefit of humankind.
Somewhat oddly, after hundreds of outstanding scientific
accomplishments for the great benefit of humankind, 1947 witnessed the launch
of the Doomsday Clock.
"And only on this solidified, granite-like foundation of
ignorance could knowledge rear itself hitherto, the will to knowledge on the
foundation of a far more powerful will, the will to ignorance, to the
uncertain, to the untrue! Not as its opposite, but—as its refinement!"
Friedrich Nietzsche.
- 5 -
Science news takes many forms and fascinates millions upon
millions.
Sometime ago, I had a look at a reputable online source that
explored a range of topics:
Lung transplantation… ancient people's height… colors
observed by birds… hazardous gases… the blood moon… a fish given engineered
nutrition… plastic pollution… and robots propelled by thinking.
The fluent reportage showed a heart warming picture:
Research reveals the fundamental truths of the physical and
biological world around us.
Scientists discover essential facts about contemporary and
past human lives.
Scientists develop innovative technologies that are vastly
beneficial to our advanced society.
And, of course, scientists expose and attempt to mitigate the
immense dangers to the environment.
Somewhat oddly, throughout the entire collection, viewed on
August 4, 2018, there was no mention of any of the troubling issues that
surround science.
Perfect silence met awkward matters like the dangers of
scientific development, hazardous technologies, and a growing variety of global
catastrophic threats.
To me, the entire collection looked like a compelling public
relations product.
Of course, you can google a bit and see for yourself if, or to what extent, a certain editorial stance shapes the gratifying science news you get.
True, the ethical code of fair reporting and balanced information is said to be strictly obliging.
But suffice it to say, that is not always the case.
The positive bias is remarkably evident when it comes to
pollution.
Science news regularly covers current scientists who identify
the dangers, raise the alarms, and offer solutions.
The other side of the picture is oddly missing.
Almost needless to add, it was past scientists who developed,
tested, authorized, promoted, and received substantial rewards for the
promising technologies, which eventually induced myriad types of global
pollution.
If you suspect an established media tradition of partisan
narratives and assorted half-truths, then you might echo the old question of
Shakespeare's Hamlet:
Is there something fishy in the regular science news columns?
Apropos, I recently looked at some well-known websites that cover pollution. The singular impact of modern scientific development on the colossal escalation of global pollution emission was barely mentioned, if at all. Could that be merely episodic, or rather intriguing?
Certain phrasing habits are no less interesting.
If you watch a TV program on the environment, you may
casually come across a scientist mentioning that pollution is “human-induced.”
Nothing more specific. Nothing like technology-induced,
regulation-induced, or business-induced. And nothing that would touch on the
embarrassing chronology: Pollution has never threatened humanity's survival
before the modern breakthroughs in science and technology.
Nonetheless, this pair of words, "human-induced,"
is pretty popular among the champions of "progress" who thrive in the
circles of the "well-educated."
As it seems, the general trend is to downplay the role of
science in the dangerous global state of affairs and, even more so, the role of
scientists in the dangerous global state of affairs.
Somewhat oddly, this trend is consistent with traditional
ways of producing media content.
Even when they do touch on the unfortunate outcome of
scientific development, mainstream media outlets tend to cover specific dangers
like nuclear weapons, engineered pathogens, or, say, fossil-fueled engines.
Looking at things from a broader perspective is not a
preferred option.
Rarely enough, there is a mention like “the dangers posed by
science,” and even more rarely, if ever, “the dangers posed by scientists.”
Small wonder many believe that
scientists are above criticism.
Many are unaware
that global problems might and do reflect the activities and inactivities of
scientists.
Many, then, would
rather not question the ethics, prestige, power, privileges, integrity,
reliability, or competence of scientists.
- 6 -
Martin Rees, an astronomer, a physicist, and the former president of the Royal Society, has published many books and research papers.
A bit surprisingly, one of his well-known books explores the
impacts of breakthrough scientific discoveries and technologies on the
dwindling safety of life on planet Earth.
If you look for a concise and authoritative work on the
catastrophic dangers of nuclear explosives, industrial pollutants, engineered
viruses, self-replicating robots, artificially intelligent computers, molecular
machines, and worldwide communication networks, this is a kind of classic.
If you are concerned with the present existential threats of
terror, world war, mega accident, cataclysmic error, risky research, and
hazardous knowledge, this work can hardly be overlooked.
And if you seek a balanced treatment of science – by a
devoted scientist, of course – this book addresses both the positive and
negative impacts of contemporary hard science.
True, the Nobel Prize ceremonies annually celebrate
outstanding contributions for the benefit of humanity. But as far as science is
concerned, “benefit” is not usually the whole story.
Science has long been compared to a double-edged sword.
And yet, the symmetry is questionable, if not optimistically
misleading.
Presently, for instance, the positive effects of science on
medical care are remarkable.
However, the negative effects on humanity's survival are undeniable, as evidenced by various indicators, such as the current position of the Doomsday Clock.
A devoted scientist like Martin Rees is clearly not an
adversary of the entire scientific endeavor. (How could he be?) Nor does he
reject the basic premise of the rationality of scientific pursuits. Indeed,
some call him an eminent defender of science in the face of its chilling
impacts.
Still, he gives a stark picture of how global catastrophic
threats induced by scientific advancements grew and will further grow.
Moreover, he observes an awkward polarization:
“Almost any applicable discovery has a potential for evil as
well as for good.”
If you conclude that the potential danger of scientific progress
keeps growing, you may not be surprised by Martin Rees' professed pessimism. Plainly,
the title of his book speaks for itself:
Our Final Century.
One of his best-known warnings – intricately phrased and
occasionally misquoted – amounts to a chilling forecast:
"I think the odds are no better than fifty-fifty that
our present civilisation on Earth will survive to the end of the present
century."
Other well-known forecasts are no less appalling.
While Frank Fenner
suggested a 100% chance of human self-extinction, Ray Kurzweil “optimistically”
implied that the ghastly chance is lower than 50%.
Nick Bostrom and
Anders Sandberg circulated a famous forecast of a 19% chance of human self-extinction
in this century due to a range of technological factors.
It follows, by the way, that the chance of a catastrophe of
"merely" a few billion human fatalities is by far higher than 19%.
If you need another tip, the chance of an old-style Russian
roulette player smashing his own head is around 16%.
Moreover, large-scale catastrophic events are typified by a
peculiar catch:
Even a small chance indicates a big threat.
That is based on formal criterions (briefly reviewed in the
References ahead.)
Indeed, given such criterions, one may reasonably argue that
taking even a 1% chance of human self-extinction – or even any non-zero chance
– is flatly unacceptable.
This or that way, it is pretty difficult to see how the human
species will survive considering the following:
- 7 -
Modern society, dependent on expertise and committed to
progress, has long boosted scientific education, empowered scientific
authority, bolstered scientific research, and enhanced the public image of
science.
Nonetheless, certain dangerous developments have drawn some
specific attention.
In 1913, the English author H.G. Wells, who was quite
familiar with the science of his day, gave a visionary description of a nuclear
weapon in a new book with a strange name: The World Set Free.
One of his close acquaintances, Winston Churchill, addressed the
growing threats of scientific progress in a 1924 article:
Shall We All Commit Suicide?
In 1933, when Leo Szilard first conceived a viable chain
reaction that could create a nuclear explosion, he was seriously alarmed and
attempted to shield the information.
But in 1939, when further research conducted in Germany
became widely known, he discussed it with Albert Einstein, who signed a letter
to President Roosevelt about the looming nuclear arms race with the Nazi
regime.
During World War II, many scientists, including current and
future Nobel laureates, participated in the Manhattan Project's production of
the nuclear bombs, which eventually wiped out Hiroshima and Nagasaki by the
order of President Truman.
However, when peace came, a few veterans founded a perturbing
periodical named Bulletin of the Atomic Scientists.
One of their close supporters was Alexander Langsdorf whose
wife, Martyl, designed the cover icon of the Bulletin in 1947. It was a piece
of graphic work called the Doomsday Clock, which showed 7 minutes before
midnight.
Later on, the displayed time was periodically altered to
reflect shifts in the immediacy of the ultimate catastrophic threat.
Right now, at the time of this writing, the clock shows 85
seconds before midnight, with all that implies.
Some historical ironies seem clear enough:
Several scientists aimed to inform, influence, and educate
politicians and common citizens about the doomsday dangers of new bombs, which
they themselves had participated in the making of.
Indeed, Robert Oppenheimer (who headed the key Los Alamos
laboratory of the Manhattan Project and later chaired the Bulletin's board of
sponsors) anticipated that one day people would curse Los Alamos, and that all
humans would vanish unless they fully cooperate.
One veteran of the Manhattan Project, Eugene Rabinowitch,
longtime editor of the Bulletin, expected more “presents” from the “Pandora's
box of modern science."
Tip:
“Nuclear weapons give an attacking nation a devastating
advantage over any feasible defense. New sciences will soon empower small
groups, even individuals, with similar leverage over society.”
Martin Rees.
- 8 -
In 2016, an anonymous Australian patient successfully
underwent a delicate surgical process. There was not much news about it, except
that the patient was a 4-year-old who swapped genders.
No one disclosed if a boy had just become a girl, or vice
versa. Either way, it was not the first nor the last case. Australian officials
confirmed that hundreds of kids had been enrolled in gender transition
programs.
The sweeping swapping trend followed a strange story.
Way back in 1907, Leo Baekeland, a chemist, invented a new
type of synthetic plastic. This material, named Bakelite, was one of the game
changers of the 20th century.
It was soon followed by PVC, nylon, and other synthetic
materials, commonly produced by the petro-chemical industries from fossil
fuels.
Media reporters, business leaders, and prominent scientists
applauded the innovative plastics, which smoothly conquered the world. The
Second World War gave a nice push to the boom, which went on for decades before
some nasty side effects drew serious attention.
It is a bit of a mystery why so many regulatory bodies around
the world dragged their feet for so long.
But better later than never, a few awkward facts at last emerged...