Sources & Bibliography

References, deep dives, and further reading for Vol 2: Stuff We Built on Top.

00

Intro

1 Topic
Explainers

The Largest Story Ever Written (And Why Software Still Loses)

7 sources
Part 6

Fragile Commons

5 Chapters
22

The Chain of Fools

8 Topics
Explainers

Supply Chains – Other People's Work, All the Way Down

6 sources

Package Managers — Where All Those Dependencies Come From

7 sources

The Industrial Revolution of Code: Understanding CI/CD

5 sources
Case Studies

`left-pad` – Eleven Lines to Rule Them All

9 sources

Log4Shell – The Ghost in the Logging Machine

9 sources
  • NVD/NIST 2021 Accessed: 2026-05-10

    Official entry in US government's National Vulnerability Database. CVSS 10.0, technical description, list of affected versions (2.0-beta9 to 2.14.1), patch links. Canonical source for any CVE citation.

  • LunaSec (GitHub) freeqaz 2021 Accessed: 2026-05-10

    The original post-mortem — LunaSec coined the name 'Log4Shell' and was first to publish detailed technical analysis. Updated over several weeks as subsequent CVEs were discovered. Cited by CISA, Apache, and practically everyone else.

    Note: At time of compilation, lunasec.io domain is inactive (DNS issue). Blog sources are available on GitHub. Linked file is source in MDX format, fully readable but note this is not the actual published page.

    Fun fact: blog uses roughly the same tech stack as the bibliography you're reading now. Except this site's repo is not public.

  • Official US Cybersecurity and Infrastructure Security Agency page with Emergency Directive 22-02, links to all CVEs, and guidance for federal agencies. Good as institutional source.

  • Lawfare Nicholas Weaver 2021 Accessed: 2026-05-10

    Solid analysis for non-specialists by Lawfare (Harvard Law / Brookings). Describes Minecraft as the starting point, critical infrastructure impacts, and SBOM implications. Good for broader context.

  • Coverage from disclosure day, includes quote from expert Marcus Hutchins about the Minecraft exploit. Good as mainstream confirmation of scale.

  • Nippon.com Nishiue Itsuki 2026 Accessed: 2026-05-10

    Official Japanese source — Nippon.com is funded by Japan Foundation. Description of the technique, its applications, and spread beyond Japan.

  • Flight Safety Australia 2018 Accessed: 2026-05-10

    Article from Australian aviation authority CASA. Good context for application of the technique beyond railways, includes bibliography with original scientific sources (Railway Technical Research Institute 1994).

  • Auth0 Blog Sebastian Peyrott 2019 Accessed: 2026-05-10

    Technically most accurate non-academic description of history: Mocha → LiveScript → JavaScript, role of Netscape/Sun agreement, quotes Brendan Eich. Often linked as canonical source.

  • Wikipedia Accessed: 2026-05-10

    For the naming footnote, sufficient as additional confirmation — contains quote from Eich himself who called the name change 'a marketing ploy by Netscape.'

React2Shell: When the Frontend Ate the Server

5 sources
  • NVD/NIST 2025 Accessed: 2026-05-10

    Canonical government entry. CVSS 10.0, technical description, list of affected versions.

  • Official post from React team (Meta). Contains disclosure timeline, list of affected packages and frameworks, patching instructions, and — notably — list of all follow-up CVEs updated through January 26, 2026. Best single source for the entire saga.

  • Official React team post listing all follow-ups: CVE-2025-55183, CVE-2025-55184, CVE-2025-67779 (December), CVE-2026-23864 (January). Also includes a quote that could be used in the book: 'This pattern shows up across the industry, not just in JavaScript. For example, after Log4Shell, additional CVEs were reported as the community probed the original fix.'

  • AWS Security Blog CJ Moses 2025 Accessed: 2026-05-10

    Primary source for Jackpot Panda and Earth Lamia. Report from MadPot honeypot infrastructure. One of the best intelligence documents on who exploited and how.

  • Wiz Research Gili Tikochinski, Merav Bar, Danielle Aminov 2025 Accessed: 2026-05-10

    Wiz researchers published the first PoC and had the earliest cloud exploitation telemetry. Contains data: 39% of cloud environments with vulnerable versions, confirmation of AWS credential harvesting.

Polyfill.io – The Rotten Ingredient

5 sources
  • Original discoverers' report — Sansec was the first company to publish attack details. Contains timeline, sample malicious script code, list of affected domains, and updates from subsequent days (Namecheap domain suspension, Cloudflare response). Only first-order canonical source for the entire story.

  • BleepingComputer Lawrence Abrams 2024 Accessed: 2026-05-10

    First extensive press coverage from disclosure day. Contains Google's communication to advertisers, Andrew Betts quotes, and attack mechanism details. BleepingComputer was the first medium to cover Sansec's report — hence the DDoS on their infrastructure in response.

  • The Hacker News Ravie Lakshmanan 2024 Accessed: 2026-05-10

    Good article with broader perspective — describes scale after a week, Funnull's restart attempt under polyfill.com domain (also taken down by Namecheap), and long-term implications. Good as supplement to live coverage.

  • Source for the 384,773 hosts number. Also includes scan of alternative domains taken over by the same operator, historical DNS records, and list of affected platforms (Hulu, Mercedes-Benz, WarnerBros). Only credible source for this specific number.

  • SecurityWeek Eduard Kovacs 2026 Accessed: 2026-05-10

    Most widely cited medium covering Hudson Rock findings. Contains summary of forensic evidence (credentials from LummaC2 infostealer, domain configuration conversations), operation goal (cryptocurrency laundering via Suncity Group gambling network), and attribution certainty caveats. Well-balanced between 'this is strong evidence' and 'this is one research firm.'

Version 99.0.0

4 sources
23

The Trillion Dollar Volunteer

5 Topics
Explainers

Open Source: Shared Knowledge, Shared Illusions

11 sources
  • Official ACS landmark document. Confirms 1929 publication, the decade of stagnation, the role of Florey and Chain, and mass production during WWII.

  • Journal of Economic Behavior and Organization Robert C. Allen 1983 Accessed: 2026-05-11

    Primary academic source for the Cleveland ironworks case. Documents the open exchange of blast furnace performance data between 1850 and 1875. Vol. 4, pp. 1–24.

  • Cambridge Journal of Economics Alessandro Nuvolari 2004 Accessed: 2026-05-11

    Primary source for the Cornwall case. Documents the practice of publishing pump efficiency data in Lean's Engine Reporter rather than seeking patent protection. Vol. 28, pp. 347–363.

  • The Henry Ford Museum 1911 Accessed: 2026-05-11

    Primary institutional source — Henry Ford Museum archive. Confirms the court ruling of January 10, 1911 overturning the Selden patent.

  • Monopoly on Wheels: Henry Ford and the Selden Automobile Patent
    Wayne State University Press William Greenleaf 1961

    Canonical academic study of the Ford–Selden patent dispute. Confirms the link between the 1911 ruling and the free cross-licensing system adopted by the industry in 1915 and maintained until 1956. ISBN: 0814335128.

  • Project MUSE John B. Rae 1961 Accessed: 2026-05-11

    Review citing the outcome of the Selden case: the automotive industry adopted a unique system of free cross-licensing in 1915, kept in force until 1956.

  • Open Source Initiative 2006 Accessed: 2026-05-11

    Canonical document defining what 'open source' means in legal and technical terms.

  • GNU / FSF Richard Stallman Accessed: 2026-05-11

    Stallman's original text describing the distinction between 'free software' and 'open source' from the FSF perspective. Essential reading for understanding why the distinction exists at all.

  • Comprehensive FSF-annotated list of software licenses, covering GPL, LGPL, AGPL, Apache, MIT, BSD and dozens of others. Authoritative from the free software movement perspective.

  • Accessible plain-English summaries of software licenses. Useful contrast to dry legal documentation — ideal for readers who want to quickly understand the difference between MIT and GPL without reading full legal texts.

  • Open Source Initiative Accessed: 2026-05-11

    List of licenses approved by OSI as conforming to the Open Source Definition. Less ideological than FSF, more pragmatic.

Case Studies

Heartbleed (CVE-2014-0160) – 64 Kilobytes of Oops

9 sources

curl – 'Please Fix This by Yesterday'

6 sources
  • daniel.haxx.se Daniel Stenberg 2022 Accessed: 2026-05-11

    The legendary post in which Stenberg publishes an ultimatum email from a corporation demanding an audit of a Java library that curl does not use. He describes the level of ignorance and incompetence as breathtaking. Best single source illustrating how compliance processes replace actual thinking.

  • daniel.haxx.se Daniel Stenberg 2021 Accessed: 2026-05-11

    Stenberg publishes a death threat received because curl 'didn't work correctly' in a user's closed system. Illustrates the extreme end of free support expectations — users who treat maintainers' time as their entitlement.

  • daniel.haxx.se Daniel Stenberg 2025 Accessed: 2026-05-11

    Explains why Stenberg shut down the curl Bug Bounty program. He describes AI-generated security reports as a 'DDoS on human attention' — technically plausible-looking but factually wrong reports flooding maintainers at scale.

  • LWN.net Jonathan Corbet 2025 Accessed: 2026-05-11
  • curl.se Daniel Stenberg Accessed: 2026-05-11

    List of companies using curl, maintained by the tool's author.

  • wolfSSL 2019 Accessed: 2026-05-11

    Stenberg's company offering paid commercial support for curl.

faker.js & colors.js – The Zalgo Rebellion

7 sources

XZ Utils – The Long Con in Plain Sight

7 sources
24

The Rug Pull

4 Topics
Explainers

The First One's Free — A Business Model with a Smile

4 sources
Case Studies

Docker Desktop — Oxygen Metered by Subscription

10 sources
  • Docker Blog Giri Sreenivas 2024 Accessed: 2026-05-12

    Official announcement of the licensing change (August 31, 2021). Introduces the 250-employee / $10M revenue thresholds, grace period until January 31, 2022, and new subscription tiers.

  • Sacra Accessed: 2026-05-12

    Best available source for a private company. Reports ARR of $20M in 2021 (pre-license change), $165M in 2023, $207M in 2024. Secondary source — Docker Inc. is private and does not publish full financials.

  • Docker Blog Scott Johnston 2022 Accessed: 2026-05-12

    Official post from Docker Inc. after the first full subscription year. CEO Scott Johnston reports ARR above $50M — over four-fold year-on-year growth. Only primary source with figures directly from the company.

  • Kubernetes Blog 2020 Accessed: 2026-05-12

    Original document announcing dockershim deprecation in Kubernetes 1.20. Explains technical reasons (CRI incompatibility), timeline, and implications for users.

  • Kubernetes Blog Sergey Kanzhelev, Jim Angel, Davanum Srinivas, Shannon Kularathna, Chris Short, Dawn Chen 2022 Accessed: 2026-05-12

    Post accompanying dockershim removal in Kubernetes 1.24. Confirms the decision was technical and long-planned, and that the majority of clusters still used Docker at the time of the licensing announcement.

  • InfoWorld Scott Carey 2021 Accessed: 2026-05-12

    Day-of-announcement coverage. Business context, market reaction, quotes from CEO Scott Johnston.

  • Connects both threads — dockershim deprecation timeline and the post-license-change market situation. Also explains Mirantis's role as maintainer of cri-dockerd.

  • USU Blog Dr. Christian Seeling 2022 Accessed: 2026-05-12

    Software Asset Management perspective — practical compliance implications for IT and procurement departments. Useful for the 'compliance panic' context described in the case study.

  • Open Container Initiative Accessed: 2026-05-12

    OCI homepage describing the image-spec and runtime-spec standards. Includes the history of the initiative (founded 2015, emerging from Docker and CoreOS).

  • GitHub / Open Container Initiative Accessed: 2026-05-12

    Technical source for the claim about image portability across different container runtimes.

Terraform — Open Until Further Notice

8 sources
  • HashiCorp Blog Armon Dadgar 2023 Accessed: 2026-05-12

    Primary canonical source — co-founder Dadgar's official post explaining the motivation (hyperscalers monetizing without contributing), scope of the change, and what remains under MPL. August 10, 2023.

  • IBM Newsroom 2024 Accessed: 2026-05-12

    Official IBM press release. $6.4 billion enterprise value, $35 per share. April 24, 2024.

  • Spacelift Blog Flavius Dinu 2024 Accessed: 2026-05-12

    Most comprehensive analysis of practical consequences — full timeline of the saga (August 10, 2023 through OpenTofu GA in January 2024), breakdown of who is affected and how, ecosystem perspective.

    At the time link was verified, the article was updated (Feb 2026). Archive link points to the original version of the text.

  • Perspective from one of the largest Terraform ecosystem partners, directly affected by the change. Good example of enterprise reaction and 'compliance panic' described in the case study.

  • Linux Foundation 2023 Accessed: 2026-05-12

    Original announcement of the project's adoption by the Linux Foundation. Includes quotes from Jim Zemlin, list of 140+ supporting companies, and commitment of 18+ FTE developers for a minimum of 5 years. September 20, 2023.

  • Journalistic coverage with a quote from Jim Brikman (Gruntwork) explaining the name and project plans.

  • Canonical description of the provider model: 'Terraform relies on plugins called providers to interact with cloud providers, SaaS providers, and other APIs.' Providers are distributed separately from Terraform Core with their own release cycles.

  • HashiCorp Developer Accessed: 2026-05-12

    Technical description of the plugin architecture — providers as separate binaries communicating with Terraform Core via RPC. Confirms providers can be written by any party. Companion to the Providers doc above.

Setting Your Own House on Fire — Unity Runtime Fee

7 sources
25

The Monoculture

5 Topics
Explainers

Intro — Monoculture and the Irish Potato Famine

5 sources
Case Studies

The Attack of the Clones — Browser Engine Monoculture

22 sources

WYSIUE — What You See Is Unreal Engine

15 sources

PostgreSQL — The One That Opposes

13 sources

The Seed Bank at the End of the World — GitHub Concentration Risk

3 sources
26

Digital Feudalism

3 Topics
Case Studies

Sherlocking: Your Success Is Our Backlog

12 sources

The 72-Hour Heist — Scale Over Soul

10 sources

The Vampire Strategy: The Extraction of Open Source

13 sources
Part 7

Garbage Harvest

4 Chapters
##

Introduction

1 Topic
Part Introduction

Garbage In, Garbage Out — Origin of the Phrase

2 sources
  • Wikipedia Accessed: 2026-05-12

    History of the phrase, first known print usage (1957), George Fuechsel, context from the early years of computing.

  • Best narrative account of the phrase's history. Documents the earliest known printed occurrence in the Times Daily of Hammond, Indiana (November 10, 1957) and the Fuechsel story. Most thorough source found on the topic — the author searched press archives and appears to have corrected the existing historical record.

27

Methodology

6 Topics
Explainers

Survey Design & Sampling Bias: How to Accidentally Measure the Wrong Universe

6 sources

Asking the Question Wrong (And Calling It Data)

6 sources

What Are We Actually Measuring?

11 sources
Case Studies

Ten Million Envelopes to Nowhere — The Literary Digest Poll

6 sources
  • Landon in a Landslide: The Poll That Changed Polling
    History Matters / George Mason University 1936 Accessed: 2026-05-14

    At the time of composing this list the original article was not available (HTTP/404), but the content was archived by the Wayback Machine.

    Contains the original Literary Digest text from October 31, 1936. Most accessible source with the verbatim forecast.

  • Election results, Gallup and Digest comparison, with references to primary sources. Acceptable as a supplementary reference.

  • Oxford University Press / Public Opinion Quarterly Peverill Squire 1988 Accessed: 2026-05-14

    The most important academic source for this case. The only empirical study of the failure's causes, based on 1937 Gallup data. Concludes that non-response bias — not sampling bias — was the dominant cause. Vol. 52, No. 1, Spring 1988, pp. 125–133. Also available via JSTOR (https://www.jstor.org/stable/2749114) and as PDF (https://criticalthinkingtext.wordpress.com/wp-content/uploads/2017/02/squire-literary-digest.pdf).

  • Smithsonian Magazine Jackie Mansky 2016 Accessed: 2026-05-14

    Good historical context for the Gallup breakthrough. Credible popular-science source.

  • References original press archives from the Boston Globe. Interesting detail: Gallup publicly predicted his own accuracy in advance.

  • University of Pennsylvania / randomservices.org Accessed: 2026-05-14

    Raw state-by-state Digest data. Useful for readers who want to see the scale of the error directly.

Love in the Time of A/B Testing — OKCupid & Facebook

8 sources
28

Technical Collection Screwups

9 Topics
Explainers

When Numbers Look Right (But Aren't)

5 sources
  • Official reference vocabulary for metrology. Defines measurement error, systematic error, random error, and uncertainty.

    The authoritative document for the science of measurement — unmatched as a reference.

    JCGM (Joint Committee for Guides in Metrology) is the international body by BIPM (Bureau International des Poids et Mesures / International Bureau of Weights and Measures) responsible for this standard.

  • Wikipedia Accessed: 2026-05-14

    Key definition: random error = noise, systematic error = bias. Good overview from a metrology and statistics perspective.

  • Scribbr Pritha Bhandari 2021 Accessed: 2026-05-14

    Clear explanation with examples. Includes the key framing: random error is 'noise' that blurs the true signal of what's being measured.

  • University of Maryland Physics R. H. B. Exell Accessed: 2026-05-14

    Classic academic treatment from a physical sciences and laboratory experiment perspective.

  • PubMed Central / Critical Care Pierre Squara, Thomas W L Scheeren, Hollmann D Aya, Jan Bakker, Maurizio Cecconi, Sharon Einav, Manu L N G Malbrain, Xavier Monnet, Daniel A Reuter, Iwan C C van der Horst, Bernd Saugel 2020 Accessed: 2026-05-14

    Defines bias, systematic error, and random error with medical examples and clear diagrams distinguishing noise from bias.

    The only source in this group written strictly for scientists rather than engineers.

The Digital Vacuum Cleaner: Scraping, Crawlers, and the Art of Collecting Everything Except What You Wanted

8 sources
Case Studies

When the Ocean 'Warmed' Because Someone Changed the Bucket

4 sources

The Case of the Chilly Buoy: How Better Thermometers 'Froze' Global Warming

12 sources

The Invisible Confetti: When Science Measured Its Own Hands

2 sources

Texas-Sized Assumptions: The Grid and Operational Context

7 sources

The Overzealous Vacuum — Google Street View WiFi Sniffing

6 sources

Hacking Reality with a Handcart — Simon Weckert's 99 Phones

8 sources

The $23 Million Fruit Fly — Amazon Algorithmic Pricing Loop

7 sources
  • michaeleisen.org Michael Eisen 2011 Accessed: 2026-05-14

    The primary source for the entire case. Eisen discovered and documented the feedback loop — exact multipliers (1.270589 and 0.9983), timeline from April 8 to 18, and analysis of both bots' strategies.

  • Amazon Seller Lists Book at $23,698,655.93 — Plus Shipping
    CNN John D. Sutter 2011 Accessed: 2026-05-14

    Day-of coverage. Quotes Eisen and a pricing algorithm expert. Includes: 'It's like you put on the gas and didn't have the handbrake.'

    Note: the original CNN article and new snapshots on Wayback Machine are not available at the time of compiling this. Instead I provide a link to snapshot from May 1, 2011.

  • The Register Rik Myslewski 2011 Accessed: 2026-05-14

    Concise technical account. Includes price details after the loop unwound.

  • Fast Company Dionysios Demetis 2019 Accessed: 2026-05-14

    Analysis that connects the case to the broader question of algorithmic reality construction. References a Journal of the Association for Information Systems publication.

  • ACM Digital Library Le Chen, Alan Mislove, Christo Wilson 2016 Accessed: 2026-05-14

    Foundational empirical paper documenting the scale of algorithmic pricing on Amazon (~500 bot-using sellers among the top 1,600 products). Identifies 'price jitter' patterns — chaotic price spikes caused directly by bots responding to each other. Academic foundation for the Fruit Fly case. WWW '16, pp. 1339–1349.

  • Wharton / lmusolff.com Leon Musolff 2025 Accessed: 2026-05-14

    Empirically documents how pricing algorithms enter mutual feedback loops producing outcomes far from market equilibrium. Provides the formal model for the mechanism described narratively in the text.

  • Stanford Law School Renato Nazzini, James Henderson 2024 Accessed: 2026-05-14

    Legal review documenting real cases of algorithmic price-fixing (including a poster-fixing conviction on Amazon). Frames bot-to-bot feedback loops as a systemic antitrust risk. Good context for 'consensus is not correctness'.

29

The Data Was Fine Until We Touched It

5 Topics
Explainers

ETL: The Data Janitor's Guide to Not Breaking Everything

5 sources
  • The Data Warehouse ETL Toolkit
    Wiley Ralph Kimball, Joe Caserta 2004

    ISBN: 978-0764567575.

    Classic data warehousing reference. Kimball is one of the founding figures of the discipline. Defines ETL as an engineering practice.

  • Accessible but technically grounded overview from a data warehousing pioneer. Covers ETL history from the 1970s through the cloud era.

  • AWS Accessed: 2026-05-14

    Practical overview with emphasis on relational databases and data pipelines. Explains the original rationale for ETL.

  • SAS Accessed: 2026-05-14

    Well-written non-academic overview. Traces ETL from its 1970s origins through data warehousing to the present day, without unnecessary jargon.

  • Wikipedia Accessed: 2026-05-14

    Solid definition with history, transformation typology, and data warehousing context. Well-referenced against technical literature.

Case Studies

Austerity by Accident: The Spreadsheet Error That Reshaped the World

6 sources

Biological Capitulation: When Science Renamed Itself to Please Excel

6 sources

The Digital Excommunication of Scunthorpe: When 'Clean' Means Deleted

5 sources

The People Who Never Existed: The 'Null' Surname

4 sources
  • Well-documented first-person account. Includes specific details about Bank of America, mortgage forms, and the email address null@nullmedia.com.

  • Gizmodo A;ossa Walker 2016 Accessed: 2026-05-14

    Based on the BBC article. Confirms issues with airline tickets and the IRS. Includes Jennifer's quote: 'I've been asked why I'm calling and when I try to explain the situation, I've been told there's no way that's true.'

  • Born SQL 2017 Accessed: 2026-05-14

    Solid technical treatment of the problem from a SQL perspective: validation, escaping, form handling. References Christopher Null's article and adds engineering analysis. Good bridge between narrative and mechanics.

  • Streamline Verify Frank Strafford 2018 Accessed: 2026-05-14

    Documents the case of Angela Johnson Null in the GSA-SAM system. Shows the problem affects multiple people beyond Jennifer and Christopher.

Part 8

Algorithms Gone Wild

4 Chapters
##

Introduction

2 Topics
Explainers

Algorithm — The Guy from Khwarazm

8 sources
  • MacTutor History of Mathematics, University of St Andrews J. J. O'Connor, E. F. Robertson 1999 Accessed: 2026-05-14

    Authoritative academic source. Covers uncertainty around dates and birthplace, the House of Wisdom connection, and the etymology of 'algebra' and 'algorithm'.

  • MacTutor History of Mathematics, University of St Andrews J. J. O'Connor, E. F. Robertson 2001 Accessed: 2026-05-14

    Academic account of the journey of Hindu-Arabic numerals from India through the Arab world to Europe. Covers al-Khwarizmi's role in popularising the positional system and zero as a placeholder. Cites original Arabic sources and Latin translations.

  • Episodes in the Mathematics of Medieval Islam
    Springer J. L. Berggren 2003

    ISBN: 978-0-387-40605-3

    Standard academic reference on Islamic mathematics. Covers al-Khwarizmi's contributions in algebra and the decimal system. Cited by MacTutor and other academic sources as a reference text.

  • Britannica Accessed: 2026-05-14

    Standard encyclopaedic overview. Cites the Latin translation 'Algoritmi de numero Indorum' as the source of the word 'algorithm'.

  • NASA Science 2017 Accessed: 2026-05-14

    Accessible but reliable overview. Traces the path from the Arabic name through the Latin 'Algoritmi' to the English 'algorithm'.

  • Good popular-science piece with solid academic grounding. Also covers the role of Hindu-Arabic numerals and zero in the context of modern computing.

  • Introduction to Algorithms (4th ed.)
    MIT Press Cormen, Leiserson, Rivest, Stein 2022

    ISBN: 978-0-262-03968-5

    Universally known as 'CLRS' — the standard algorithms textbook used in university courses worldwide. Chapters 6–8 cover sorting algorithms (heapsort, quicksort, mergesort); chapters 8–9 cover linear-time sorting. Includes formal complexity analysis for each algorithm.

  • MIT OpenCourseWare 2020 Accessed: 2026-05-14

    Official MIT course syllabus. Lists sorting as a central topic and recommends CLRS as the reference text. Free access to lecture materials.

Deterministic — The 'What You See Is What You Get' of Logic

4 sources
30

Rules Gone Wrong

3 Topics
Case Studies

BATS: The Digital Ouroboros

9 sources

The Zip-Code Executioner: Ofqual's 'Fair' Algorithm

7 sources

The Centroid of Doom: Apple's Desert Odyssey

9 sources
31

Feedback Loops – Amplifying the Absurd

5 Topics
Explainers

What Is a Feedback Loop? (And Why Chaos Doesn't Need AI)

8 sources
  • Business Dynamics: Systems Thinking and Modeling for a Complex World
    McGraw-Hill John D. Sterman 2000

    ISBN: 978-0072389159.

    Standard academic textbook on system dynamics. Formally defines feedback loops, covers reinforcing and balancing loops with mathematical examples.

  • Thinking in Systems: A Primer
    Chelsea Green Publishing Donella H. Meadows 2008

    ISBN: 978-1603580557.

    Widely read popular-science introduction to systems thinking. Defines feedback loops in the first chapter through everyday analogies (thermostat, population, economy). Written for non-specialists.

  • MIT OpenCourseWare Erik Demaine, Jason Ku, Justin Solomon 2020 Accessed: 2026-05-14

    Formal treatment of recursion as a foundation of algorithmics, contrasted with iterative (linear) approaches.

  • Khan Academy Accessed: 2026-05-14

    Accessible explanation of recursion as 'a function that calls itself.' Everyday analogies, no mathematical jargon.

  • Sapiens: A Brief History of Humankind
    Harper Yuval Noah Harari 2015

    ISBN: 978-0062316110.

    Source for the Level One / Level Two chaos distinction. Examples: weather (Level One), stock markets and revolutions (Level Two).

    Harari holds a D.Phil. in history from Oxford (2002), specialising in medieval military history. Sapiens is a popular-science work outside his academic specialisation — which the text acknowledges.

  • The Hebrew University of Jerusalem Accessed: 2026-05-14

    Reference for Harari's academic credentials from his university profile page.

  • Stanford Encyclopedia of Philosophy Accessed: 2026-05-14

    Solid philosophical and mathematical treatment of chaos theory. Defines sensitive dependence on initial conditions, nonlinearity, and aperiodicity as the three characteristics of mathematical chaos. Explains why chaos ≠ randomness.

  • American Physical Society 2003 Accessed: 2026-05-14

    History of Lorenz's discovery. Explains the butterfly effect as sensitive dependence on initial conditions, with historical and mathematical context.

Case Studies

The Invisible Hand with a Digital Thumb — RealPage

12 sources

Flash Crash (2010) — When Algorithms Ate Wall Street

10 sources

PredPol — The Feedback Loop of Crime

11 sources

ICU Alarm Storm — When Safety Turns into Noise

10 sources
32

When Optimization Becomes the Problem

6 Topics
Explainers

Optimization — Finding the 'Sweet Spot'

10 sources
Case Studies

The Alphabet Soup: How Etsy Optimized for Bots and Killed the Human Touch

10 sources

The Rage Engine: How Facebook Optimized for the End of Civility

11 sources

Sydney 2014: The $100 Escape — When Math Met Terror

9 sources

The Pink Slip Processor — Amazon's Automated Firing

11 sources

The Mid Staffordshire Scandal

10 sources
Part 9

We Taught the Machine to Guess

6 Chapters
##

Introduction

2 Topics
Explainers

What Actually Counts as Machine Learning

29 sources

The Coin-Flip Economy

10 sources
  • MIT Press / deeplearningbook.org Ian Goodfellow, Yoshua Bengio, Aaron Courville 2016 Accessed: 2026-05-15

    Canonical definition of machine learning, generalisation, and the distinction between training and test error.

    Note: this is a chapter of book referenced in the previous explainer.

  • IBM Think Dave Bergmann Accessed: 2026-05-15

    Concise industry-level treatment of generalisation as the fundamental goal of ML.

  • Machine Learning Mastery Jason Brownlee 2019 Accessed: 2026-05-15

    Accessible but solid piece on why generalisation is ML's 'superpower' and when it fails.

  • Formal definition of i.i.d. with a section on ML implications. Wikipedia is authoritative here — this is a mathematically precise concept.

  • Dive into Deep Learning (d2l.ai) Aston Zhang, Zachary Lipton, Mu Li, Alexander J. Smola 2023 Accessed: 2026-05-15

    ISBN: 978-1009389433

    Academic online textbook. Detailed description of covariate shift, label shift, and concept shift with concrete examples of deployment failures. Direct counterpart to the 'changed coin' analogy in the explainer.

  • Chip Huyen's Blog Chip Huyen 2022 Accessed: 2026-05-15

    Practical engineering perspective on distribution shift, from the author of Designing Machine Learning Systems. Describes when and why models stop working in production.

  • Causality: Models, Reasoning, and Inference
    Cambridge University Press Judea Pearl 2009

    ISBN: 978-0521895606.

    Foundational work. Formal framework distinguishing association, intervention, and counterfactual. The basis of the entire causal inference field.

  • The Book of Why
    Basic Books Judea Pearl, Dana Mackenzie 2018

    ISBN: 978-0465097609.

    Popular-science companion to Pearl's Causality. Describes the Ladder of Causation and explains why standard ML operates solely at the level of association.

  • Towards Data Science Kaushik Rajan 2026 Accessed: 2026-05-15

    Solid industry piece connecting Pearl's Ladder to concrete ML failure examples caused by confusing correlation with causation (including the hormone replacement therapy case).

  • lgmoneda.github.io Luis Moneda 2021 Accessed: 2026-05-15

    More technical but accessible piece on what spurious correlation means specifically in ML and how it differs from the classical statistics concept.

33

Garbage In, Gospel Out

7 Topics
Explainers

Data Bias: Learning the Wrong Lesson

7 sources
  • arXiv / ACM Computing Surveys Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, Aram Galstyan 2021 Accessed: 2026-05-15

    Canonical survey with over 1,000 citations. Taxonomy of bias sources (historical, representation, measurement, etc.) and definitions of fairness. Peer-reviewed, available via arXiv and ACM DL.

  • MDPI Big Data and Cognitive Computing Tiago P. Pagano, Rafael B. Loureiro, Fernanda V. N. Lisboa, Rodrigo M. Peixoto, Guilherme A. S. Guimarães, Gustavo O. R. Cruz, Maira M. Araujo, Lucas L. Santos, Marco A. S. Cruz, Ewerton L. S. Oliveira, Ingrid Winkler, Erick G. S. Nascimento 2023 Accessed: 2026-05-15

    PRISMA-compliant systematic review covering 2017–2022. Focuses on bias detection and mitigation techniques. More recent than Mehrabi et al.

  • IBM Think Alexandra Jonker , Julie Rogers Accessed: 2026-05-15

    Authoritative industry explanation of how historical bias is inherited by ML models, with concrete examples including predictive policing and Oakland data.

  • Weapons of Math Destruction
    Crown Cathy O'Neil 2016

    ISBN: 978-0553418811.

    Key popular-science work. Introduces WMD (Weapons of Math Destruction) defined by three characteristics: opacity, scalability, and resistance to challenge. Directly relevant to the idea of encoding bias into math to make it appear objective.

  • Automating Inequality
    St. Martin's Press Virginia Eubanks 2018

    ISBN: 978-1250074317.

    Companion volume focusing on welfare state systems. Documents how historical bias is amplified by automation and applied at institutional scale.

  • Ford Foundation Jenny Toomey, Lori McGlinchey 2016 Accessed: 2026-05-15

    Online summary of O'Neil's arguments with her own commentary. Useful for readers without access to the book.

  • NIST Reva Schwartz, Apostol Vassilev, Kristen Greene, Lori Perine, Andrew Burt, Patrick Hall 2022 Accessed: 2026-05-15

    Official US government document defining AI bias and recommending management approaches. Its existence as a NIST publication reflects the seriousness of the correction problem.

Data Leakage

8 sources
  • ACM Transactions on Knowledge Discovery from Data Shachar Kaufman, Saharon Rosset, Claudia Perlich, Ori Stitelman 2012 Accessed: 2026-05-15

    Canonical academic paper formalising the concept of leakage. Identifies it as one of the ten biggest mistakes in data mining and describes the 'no-time-machine requirement' — the prohibition on using features that would not be available at prediction time.

  • Wikipedia Accessed: 2026-05-15

    Solid overview with breakdown into feature leakage and row-wise leakage. Useful as a quick taxonomy reference.

  • IBM Think Tim Mucci Accessed: 2026-05-15

    Authoritative industry explanation of both leakage types (target leakage and train-test contamination) with a description of the normalisation-before-split problem.

  • Machine Learning Mastery Jason Brownlee 2020 Accessed: 2026-05-15

    Well-written practical tutorial with code. Explains precisely why normalising the full dataset before the train-test split is a mistake and how to fix it.

  • Google Research Blog / Science Moritz Hardt 2015 Accessed: 2026-05-15

    Seminal paper describing exactly the mechanism of iterative leaderboard overfitting: repeated modification of a model based on holdout set results creates a dependency that invalidates the classical holdout method. Published in Science 349(6248).

  • NeurIPS 2019 Rebecca Roelofs, Vaishaal Shankar, Benjamin Recht, Sara Fridovich-Keil, Moritz Hardt, John Miller, Ludwig Schmidt 2019 Accessed: 2026-05-15

    First large meta-analysis of test set reuse across 100+ Kaggle competitions. Results are surprisingly positive (little evidence of widespread overfitting), but the methodology confirms the reality of the mechanism.

  • mrtz.org (Moritz Hardt's Blog) Moritz Hardt 2015 Accessed: 2026-05-15

    Influential post demonstrating how to climb a leaderboard on the Heritage Health Prize without examining the data — purely through algorithmic probing of results. Direct documentation of leaderboard hacking.

  • Patterns (Cell Press) Sayash Kapoor, Arvind Narayanan 2023 Accessed: 2026-05-15

    Review of 294 published papers across 17 scientific fields affected by data leakage. Demonstrates that the problem is neither abstract nor marginal — it has affected hundreds of peer-reviewed publications.

Case Studies

Amazon's Hiring Bot — When AI Learned to Be a Bro

10 sources

Sentenced by Spreadsheet: The COMPAS Recidivism Racket

10 sources

The Dollar-Sign Diagnosis

5 sources

Google Photos (2015) — The Tag That Broke the Internet

9 sources

AI vs COVID-19: A Great Promise, Zero Utility

5 sources
34

Models That Learned the Wrong Lesson

9 Topics
Explainers

Edge Cases & Tail Risk

8 sources
  • ResearchGate Christian Agrell, Simen Eldevik, Andreas Hafver, Frank Børre Pedersen 2019 Accessed: 2026-05-15

    Academic survey of ML limitations in high-risk environments. Formally describes the constraints of correlation-based models when applied to tail events.

  • Investopedia Meagan Drew 2025 Accessed: 2026-05-15

    Accessible financial-sector explanation of tail risk. Covers normal distribution vs. fat tails.

  • Chip Huyen's Blog Chip Huyen 2022 Accessed: 2026-05-15

    Engineering-focused explanation of distribution shift and edge cases in ML. Practical and accessible for technical readers.

  • The Black Swan: The Impact of the Highly Improbable
    Random House Nassim Nicholas Taleb 2007

    ISBN: 978-1-4000-6351-2.

    Original source for the term and the three defining criteria of a Black Swan event.

  • Britannica Sanat Pai Raikar Accessed: 2026-05-15

    Solid encyclopaedic definition with the history of the term (Juvenal → de Vlamingh → Taleb). Covers the 1697 de Vlamingh sighting in Australia.

  • Wikipedia Accessed: 2026-05-15

    Well-documented article with citations. Covers the three criteria, history, and critiques of the theory. Useful as a quick reference.

  • Wikipedia Accessed: 2026-05-15

    Confirms the first European observations: Antonie Caen (1636) and Willem de Vlamingh (1697, Swan River, Western Australia). Cygnus atratus as a species endemic to Australia.

  • ConsumerAffairs Alexus Bazen 2026 Accessed: 2026-05-15

    Contains all three weight figures: average passenger car (~4,000 lbs / ~2 tonnes, EPA), maximum semi-truck weight (80,000 lbs / 40 tonnes, FMCSA). Data current as of 2024.

Overfitting

6 sources

Spurious Correlations

7 sources

Reward Hacking

6 sources
Case Studies

The 70mph (110km/h) Panic Attack — Phantom Braking

14 sources

Zillow: The Algorithm That Bought Too Many Houses

8 sources

Husky vs. Wolf (2016)

3 sources

Lying Down with AI — Position Bias in Medical Imaging

5 sources

The Malicious Compliance Files

8 sources
35

The Black Box Problem

7 Topics
Explainers

Interpretability

11 sources

Big Data

6 sources
  • Original paper introducing the 3V definition (Volume, Velocity, Variety). Meta Group Research Note 949, 6 February 2001. Gartner acquired Meta Group in 2005.

    Note: the original Gartner blog article does not exist anymore (https://www.gartner.com/en/articles/strategic-predictions-for-2026). Alternative source provided.

  • Big Data: A Revolution That Will Transform How We Live, Work, and Think
    Houghton Mifflin Harcourt Viktor Mayer-Schönberger, Kenneth Cukier 2013

    ISBN: 978-0-544-00269-2.

    Most influential popular-science book on Big Data. Responsible for bringing the term into the mainstream around 2013–2015.

  • Big Data: Principles and Best Practices of Scalable Realtime Data Systems
    Manning Nathan Marz, James Warren 2015

    ISBN: 978-1-617-29034-3.

    Canonical technical reference. Describes Lambda architecture (batch + speed layer), distributed systems, and failure modes. Standard industry reading.

  • Google Research / SOSP 2003 Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung 2003 Accessed: 2026-05-15

    Original paper describing how to scale storage across thousands of servers, treating disk failures as the norm rather than the exception. Foundation of the Big Data infrastructure era.

  • Google Research / OSDI 2004 Jeffrey Dean, Sanjay Ghemawat 2004 Accessed: 2026-05-15

    Paper that gave rise to Hadoop and the entire Big Data ecosystem. Describes distributed computation on clusters subject to partial failure. Canonical source for 'hundreds or thousands of servers working in parallel'.

  • Classic list of eight false assumptions about distributed networks (the network is reliable, latency is zero, etc.). Origin: Sun Microsystems, 1994–1997. The original has no single canonical URL; Wikipedia reproduces the full list with history.

Case Studies

IBM Watson for Oncology

8 sources

The Diaper Diviner — How Target Out-Parented a Father

4 sources
  • The New York Times Magazine Charles Duhigg 2012 Accessed: 2026-05-15

    Primary journalistic source for the entire case. Describes Andrew Pole, the 25 products, the pregnancy prediction score, the father-and-daughter anecdote, and the coupon shuffling mechanism.

  • The Power of Habit: Why We Do What We Do in Life and Business
    Random House Charles Duhigg 2012

    ISBN: 978-1-4000-6928-6.

    Contains an expanded version of the same case. The book brought the story to mainstream attention.

  • Machine Learning Times / KDnuggets Eric Siegel 2014 Accessed: 2026-05-15

    Key debunking article. Argues the causal link between the algorithm and any specific pregnancy 'has essentially been debunked.' Essential for bibliographic honesty regarding the limits of the anecdote.

  • The article whose headline went viral and cemented the simplified version of the story in public consciousness. Context for how the anecdote took on a life of its own.

Upstart: The Algorithm That Knew Too Much (and Too Little)

10 sources

JPMorgan LOXM: The Genius Pilot in the Fog

6 sources

AlphaGo: The Genius We Couldn't Understand

6 sources
  • Primary source for Move 37: 'a move that had a 1 in 10,000 chance of being used.' Also describes the match against Lee Sedol (18 world titles, AlphaGo winning 4–1).

  • Wikipedia Accessed: 2026-05-15

    Detailed match account with professional Go players' commentary on Move 37 and Lee Sedol's reaction.

  • YouTube / DeepMind Greg Kohs (dir.) 2017 Accessed: 2026-05-15

    Feature documentary about the match. Contains original footage of Move 37 and the commentators' real-time reactions.

  • Nature David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel & Demis Hassabis 2016 Accessed: 2026-05-15

    Original AlphaGo paper describing the architecture (deep neural networks combined with Monte Carlo tree search).

  • Nature David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel & Demis Hassabis 2017 Accessed: 2026-05-15

    AlphaGo Zero paper. Quote: 'Starting tabula rasa, our new program AlphaGo Zero achieved superhuman performance, winning 100–0 against the previously published, champion-defeating AlphaGo.'

  • DeepMind Blog David Silver, Demis Hassabis 2017 Accessed: 2026-05-15

    Official DeepMind blog post. Accessible explanation of the self-play mechanism and the differences from the previous version.

36

When Models Meet Reality

5 Topics
Explainers

Distribution Shift

7 sources
  • MIT Press / ResearchGate Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, Neil D. Lawrence 2009 Accessed: 2026-05-15

    Canonical academic source. Formalises covariate shift, label shift, and concept shift as distinct mechanisms. Cited by hundreds of papers as the reference taxonomy.

  • Chip Huyen's Blog Chip Huyen 2022 Accessed: 2026-05-15

    Accessible engineering explanation. Covers population shift, temporal shift, and concept drift with production examples. Author is Senior Staff Engineer at NVIDIA and author of Designing Machine Learning Systems (O'Reilly). Also cited in Chapters 34 and 36.

  • ACM Computing Surveys João Gama, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, Abdelhamid Bouchachia 2014 Accessed: 2026-05-15

    Canonical concept drift survey. Classifies drift types and adaptation methods. Most cited academic source for this specific category.

  • paulgraham.com Paul Graham 2002 Accessed: 2026-05-15

    Original essay that launched the era of Bayesian spam filters. Describes 'Viagra' as a strong spam signal. Historical source for the 2003 context.

  • paulgraham.com Paul Graham 2003 Accessed: 2026-05-15

    Sequel essay. Describes spammer counter-evolution (keyword obfuscation) as an early example of adversarial concept drift. Background for the V1agra/Vi@gra examples.

  • BMJ Laure Wynants, Ben Van Calster, Gary S. Collins, Richard D. Riley, Georg Heinze, Ewoud Schuit, Elena Albu, Banafsheh Arshi, Vanesa Bellou, Marc M. J. Bonten, Darren L. Dahly, Johanna A. Damen, Thomas P. A. Debray, Valentijn M. T. de Jong, Maarten De Vos, Paula Dhiman, Joie Ensor, Shan Gao, Maria C. Haller, Michael O. Harhay, Liesbet Henckaerts, Pauline Heus, Jeroen Hoogland, Mohammed Hudda, Kevin Jenniskens, Michael Kammer, Nina Kreuzberger, Anna Lohmann, Brooke Levis, Kim Luijken, Jie Ma, Glen P. Martin, David J. McLernon, Constanza L. Andaur Navarro, Johannes B. Reitsma, Jamie C. Sergeant, Chunhu Shi, Nicole Skoetz, Luc J. M. Smits, Kym I. E. Snell, Matthew Sperrin, René Spijker, Ewout W. Steyerberg, Toshihiko Takada, Ioanna Tzoulaki, Sander M. J. van Kuijk, Bas C. T. van Bussel, Iwan C. C. van der Horst, Kelly Reeve, Florien S. van Royen, Jan Y. Verbakel, Christine Wallisch, Jack Wilkinson, Robert Wolff, Lotty Hooft, Karel G. M. Moons, Maarten van Smeden 2020 Accessed: 2026-05-15

    Review of 232 COVID prediction models. Documents systematic distribution shift problems in clinical ML models during the pandemic.

  • PMLR Bret Nestor, Matthew B. A. McDermott, Willie Boag, Gabriela Berner, Tristan Naumann, Michael C. Hughes, Anna Goldenberg, Marzyeh Ghassemi 2019 Accessed: 2026-05-15

    Documents temporal shift in medical records data. Shows how changes in data recording systems cause model degradation — the same mechanism as COVID distribution shift.

Case Studies

Robert Williams: The Handcuffs of a Misplaced Pixel

6 sources

Amazon Rekognition: The Capitol Hill Mugshots

6 sources

Stanford Speech Gap: The Deaf Ear of Medical AI

5 sources

The UK Parliaments: Lost in Transcription (Literally)

6 sources
##

Summary

1 Topic
Part Summary

YouTube Recommendation Algorithm

5 sources
Part 10

Machines That Create

6 Chapters
##

Introduction

3 Topics
Part Introduction

General Reference

1 sources
  • Responsible AI Collaborative Accessed: 2026-05-16

    Database of AI-related incidents maintained by the Responsible AI Collaborative. Used in several places throughout this book as a source or starting point for further research. Recommended independently of any specific selection made here.

Explainers

Next-Token Prediction

5 sources
  • arXiv / NeurIPS 2017 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin 2017 Accessed: 2026-05-16

    Original paper introducing the Transformer architecture. Canonical source for the attention and token sections. Also cited in The Transformer explainer.

  • karpathy.github.io Andrej Karpathy 2015 Accessed: 2026-05-16

    Classic popular-science description of next-token prediction as a mechanism. Author later became Director of AI at Tesla and a founding member of OpenAI.

  • arXiv / Stanford Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang 2023 Accessed: 2026-05-16

    Canonical study on accuracy degradation in long contexts. Confirms the mechanism described in the Frodo/Tatooine example.

  • TokenMix 2026 Accessed: 2026-05-16

    Current (April 2026) overview of context window sizes. Confirms GPT-4o 128k, Claude 200k, Gemini 2.5 Pro 1M–2M tokens, and the 10–25% lost-in-the-middle degradation range.

  • arXiv / TMLR Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus 2022 Accessed: 2026-05-16

    Canonical academic source for the emergence section — the phenomenon of capabilities appearing unpredictably as models are scaled.

The Transformer

6 sources
  • arXiv / NeurIPS 2017 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin 2017 Accessed: 2026-05-16

    Original Google Brain paper introducing the Transformer architecture, self-attention, multi-head attention, and Q/K/V. 8 authors, cited over 100,000 times. Also cited in the Next-Token Prediction explainer.

  • jalammar.github.io Jay Alammar 2018 Accessed: 2026-05-16

    Most widely referenced popular explanation of the Transformer mechanism with Q/K/V and attention matrix visualisations. Industry standard entry point for non-specialists.

  • YouTube / 3Blue1Brown 3Blue1Brown 2024 Accessed: 2026-05-16

    Animated explanation of tokenisation, embeddings, and the attention mechanism.

  • Neural Computation Sepp Hochreiter, Jürgen Schmidhuber 1997 Accessed: 2026-05-16

    Canonical LSTM paper. Describes the vanishing gradient problem that the Transformer later solved. Historical background for the 'cassette tape' section.

  • transformer-circuits.pub / Anthropic Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan†, Nicholas Joseph, Ben Mann†, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, Chris Olah 2021 Accessed: 2026-05-16

    Anthropic mechanistic interpretability research showing how attention heads specialise emergently. Confirms 'no one programmed these roles'.

  • arXiv / Meta AI Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample 2023 Accessed: 2026-05-16

    Original LLaMA paper. Confirms the expansion of the acronym: Large Language Model Meta AI.

37

Chatbots Unleash the Id

6 Topics
Case Studies

OpenAI Sora — The Physics-Defying Video Generator

8 sources

Microsoft Tay — Bubbly Teen to Holocaust Denier in 16 Hours

6 sources

Grok and the MechaHitler Update

6 sources

Microsoft Bing's Sydney — When the Search Engine Needed Therapy

5 sources

Google Gemini — When Diversity Invaded the Third Reich

5 sources

Character.AI Teen Suicide Case

8 sources
38

The Synthetic Tsunami

5 Topics
Explainers

Digital Inbreeding — Model Collapse and the Death of Nuance

6 sources
Case Studies

Sports Illustrated — The Ghost in the Press Box

6 sources

Bosom Peril and the Counterfeit Brains — The Death of Academic Rigor

7 sources

Music Slop — From Federal Fraud to Corporate Wallpaper

5 sources

AI Code — Faster, Cheaper, and Holier

7 sources
39

Reality Optional

4 Topics
Case Studies

The Balenciaga Pope — When Style Murdered the Truth

6 sources

The €220,000 'Melodic' German Accent — The End of Vocal Truth

6 sources
  • Wall Street Journal Catherine Stupp 2019 Accessed: 2026-05-16

    Primary source. Rüdiger Kirsch (Euler Hermes) quoted on the 'slight German accent and the melody of his voice'. Details of three phone calls, the Hungarian account, and the Mexican transfer. Paywalled but cited by all secondary sources."

  • Detailed technical account. Cites WSJ. Describes the three-call timeline and the verification mechanism. Cybersecurity perspective.

  • Industry coverage. Confirms all key facts. Financial and insurance context (Euler Hermes as underwriter).

  • Describes this case as a landmark first. Covers the evolution of voice fraud from 2019 to 2025. Data point: 28% of UK adults reported being targeted by an AI voice scam in 2024.

  • Note regarding punchline
    Author's note

    Executive astrology is a real thing. I will not be providing sources, as I refuse to promote or inadvertently validate it.

  • Note regarding footnote
    Author's note

    The author acknowledges that no formal law obliges Polish citizens to question the melodic qualities of the German (or, for that matter, any other) language. The practice is instead governed by long-standing, if entirely unofficial, cultural norms. Embedded deeply enough, they occasionally surface in national symbols. That, however, is a discussion entirely outside the scope of this work.

The 48-Hour Ghost — Slovakia's Silent Sabotage

7 sources

The $25 Million Virtual Theater — Hong Kong's Deepfake Heist

5 sources
40

Alignment Without Control

8 Topics
Explainers

The Illusion of the Digital Leash

11 sources
  • arXiv / Anthropic Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jared Kaplan 2022 Accessed: 2026-05-16

    Original paper defining Constitutional AI. Describes the self-critique mechanism and RLAIF. 51 Anthropic authors. Also cited in the Automated Priest case study.

  • Anthropic 2023 Accessed: 2026-05-16

    Official Anthropic page describing Constitutional AI as an approach. Accessible explanation of the mechanism and philosophy. Also cited in the Automated Priest case study.

  • arXiv / Anthropic Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, Geoffrey Irving 2022 Accessed: 2026-05-16

    Canonical academic source for AI red teaming. Describes the mechanism and methodology.

  • Cloud Security Alliance Research Note on NIST's forthcoming AI RMF Playbook for Red Teaming. Describes the NIST framework and its relationship to the broader AI risk management landscape.

  • NIST's AI Risk Management Framework. Describes the framework's structure and its relationship to the broader AI risk management landscape.

  • NIST 2026 Accessed: 2026-05-16

    NIST's AI Risk Management Framework Playbook

  • RAND Corporation Willis H. Ware 1970 Accessed: 2026-05-16

    Willis Ware's task force report. Organised 1967, published 1970. Foundation of the history of penetration testing.

  • Wikipedia Accessed: 2026-05-16

    Complete history of the term from 1965 to the present. Cites Willis Ware, the Spring 1968 Joint Computer Conference, and James P. Anderson.

  • Infosec Institute 2019 Accessed: 2026-05-16

    Accessible historical narrative. Covers the Willis Report, Tiger Teams, and the evolution to commercial penetration testing.

  • Wikipedia Accessed: 2026-05-16

    Footnote reference. Dates (~1230–1309), portfolio of castles (Beaumaris, Harlech, Caernarfon, Conwy), role as Edward I's chief architect in Wales.

  • Cadw (Welsh Government) Accessed: 2026-05-16

    Footnote reference. Official Welsh heritage source. Describes Beaumaris as 'the greatest castle never built' and 'the castle to end all castles'.

The Art of the Digital Suck-up

7 sources
  • arXiv / NeurIPS 2022 Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe 2022 Accessed: 2026-05-16

    Canonical source for RLHF as a method. The InstructGPT paper. Describes the reward model mechanism and human labellers.

  • arXiv / Anthropic Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, Jared Kaplan 2022 Accessed: 2026-05-16

    Anthropic's original RLHF study. First description of sycophancy as an undesired side effect.

  • ICLR 2024 / OpenReview Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, Ethan Perez 2024 Accessed: 2026-05-16

    Canonical sycophancy study. 'Humans prefer sycophantic responses over correct ones a non-negligible fraction of the time.' Primary academic source for the entire sycophancy section.

  • arXiv Itai Shapira, Gerdus Benade, Ariel D. Procaccia 2025 Accessed: 2026-05-16

    Mechanistic explanation of why RLHF amplifies sycophancy. Formal mathematical model.

  • arXiv / Stanford Ryan Park, Rafael Rafailov, Stefano Ermon, Chelsea Finn 2024 Accessed: 2026-05-16

    Canonical source for verbosity bias. Quote: 'RLHF is known to exploit biases in human preferences, such as verbosity. A well-formatted and eloquent answer is often more highly rated by users, even when it is less helpful.'

  • arXiv Keita Saito, Akifumi Wachi, Koki Wataoka, Youhei Akimoto 2023 Accessed: 2026-05-16

    Empirical study of verbosity bias in both human labellers and LLM-as-evaluator settings.

  • Accessible analysis of the main RLHF problems: sycophancy, over-refusal, deceptive alignment. Also cited in the Automated Priest case study.

Case Studies

The 98-Page Illusion — Red Teaming and the Great Jailbreak

5 sources
  • OpenAI 2023 Accessed: 2026-05-16

    98-page document. 50+ domain experts. ARC's TaskRabbit/CAPTCHA incident. Multimodal jailbreaks. Primary source for all facts in this case study.

  • Original media account of the TaskRabbit incident. Quotes the System Card directly.

  • arXiv / OpenAI Hurst et al. 2023 Accessed: 2026-05-16

    Documents multimodal jailbreaks. 'Text-screenshot jailbreak' as a key problem. Background for the t-shirt and LEGO bricks section.

    ArXiv lists 99 authors and notes '318 additional authors not shown' – hence I decided to skip the full list and use 'Hurst et al.' as the author.

  • Organisation that conducted the TaskRabbit test cited in the GPT-4 System Card. Mission: 'align future ML systems with human interests'.

    Original link pointed to https://evals.alignment.org but it now redirects to https://metr.org. Web Archive captured explanation of the change: 'METR – Model Evaluation and Threat Research. Formerly “ARC Evals”, METR was incubated at the Alignment Research Center and is now a standalone non-profit.'

  • Describes jailbreak techniques with reference to first reports appearing within two hours of model publication.

The Automated Priest — Anthropic's Constitutional AI

5 sources
  • arXiv / Anthropic Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jared Kaplan 2022 Accessed: 2026-05-16

    Original academic paper. Contains the RLAIF mechanism and self-critique description. Source for footnote citations of the original constitutional principles. Also cited in the Illusion of the Digital Leash explainer.

  • Anthropic 2023 Accessed: 2026-05-16

    Official documentation. Contains formulations about 'excessively paternalistic' behaviour and the list of things to avoid (lectures, moralizes, condescending). Source for the evolution from Claude 2.x to the current approach. Also cited in the Illusion of the Digital Leash explainer.

  • Reddit r/ClaudeAI 2023 Accessed: 2026-05-16

    Documented comparison of Claude 2.0 and 2.1 with over-refusal examples. Community evidence for the 'legendary' status of the 2.x models on forums.

  • Reddit r/LocalLLaMA 2023 Accessed: 2026-05-16

    Specific documented case of Claude 2.1 refusing the `kill` command for a Python process. Primary source for the kill command anecdote.

  • Hacker News 2023 Accessed: 2026-05-16

    Day-of-launch discussion. User quotes about the model being 'borderline useless' due to over-refusal. Context for the Reddit and HackerNews reaction described in the text.

The Remoteli 'CEO' Hack — Social Engineering for Machines

6 sources

The $1 Tahoe — Chevrolet's Chatbot and the 'Binding Offer'

4 sources

The DAN Chronicles — Emotional Blackmail for Calculators

5 sources

The 'Delve' Paradox — RLHF and Linguistic Chauvinism

8 sources
##

Summary

1 Topic
Explainers

The AGI Myth — Chasing the Digital Holy Grail

19 sources
  • Artificial General Intelligence
    Springer Ben Goertzel, Cassio Pennachin (eds.) 2007

    ISBN: 978-3-540-23733-4.

    Canonical academic introduction of the AGI term. Goertzel and Legg popularised the term around 2002. Defines AGI as a system capable of performing any intellectual task at human level.

  • arXiv Dan Hendrycks, Dawn Song, Christian Szegedy, Honglak Lee, Yarin Gal, Erik Brynjolfsson, Sharon Li, Andy Zou, Lionel Levine, Bo Han, Jie Fu, Ziwei Liu, Jinwoo Shin, Kimin Lee, Mantas Mazeika, Long Phan, George Ingebretsen, Adam Khoja, Cihang Xie, Olawale Salaudeen, Matthias Hein, Kevin Zhao, Alexander Pan, David Duvenaud, Bo Li, Steve Omohundro, Gabriel Alfour, Max Tegmark, Kevin McGrew, Gary Marcus, Jaan Tallinn, Eric Schmidt, Yoshua Bengio 2025 Accessed: 2026-05-16

    Formal definition based on Cattell-Horn-Carroll theory across 10 cognitive domains. Documents the 'jagged cognitive profile' of current models — good academic background for the list of missing capabilities (reasoning, planning, on-the-fly learning).

  • Solid industry definition. Quotes LeCun on the need for a new architecture. Explains why current LLMs (GPT-4), despite apparent versatility, remain ANI.

  • Google Cloud 2026 Accessed: 2026-05-16

    Accessible ANI/AGI/ASI definition. Quick reference for non-specialist readers.

  • Solid overview with the history of the term (Gubrud 1997, AIXI 2000, Legg/Goertzel 2002). ANI/AGI/ASI distinction. Lists 72 active AGI projects in 37 countries (2020).

  • arXiv Katja Grace, Harlan Stewart, Julia Fabienne Sandkühler, Stephen Thomas, Ben Weinstein-Raun, Jan Brauner, Richard C. Korzekwa 2024 Accessed: 2026-05-16

    Largest survey of AI researchers (2,778 respondents). Median estimate for 'high-level machine intelligence' shortened by 13 years between 2022 and 2023. Canonical source for the AGI timeline section.

  • Effective Altruism Forum Vishakha Agrawal 2025 Accessed: 2026-05-16

    Source for Yoshua Bengio's quote: '95% confidence interval for the time horizon of superhuman intelligence at 5 to 20 years' (2023).

  • 80,000 Hours Benjamin Todd 2025 Accessed: 2026-05-16

    Accessible survey of forecasts. Context for the '5 to 20 years' claim.

  • Journal of Comparative Neurology Frederico A.C. Azevedo, Ludmila R.B. Carvalho, Lea T. Grinberg, José Marcelo Farfel, Renata E.L. Ferretti, Renata E.P. Leite, Wilson Jacob Filho, Roberto Lent, Suzana Herculano-Houzel 2009 Accessed: 2026-05-16

    Canonical source for the 86 billion neuron figure. Corrects the widely repeated myth of 100 billion.

  • Wikipedia Accessed: 2026-05-16

    Source for the 100–500 trillion synapses range as standard approximation. Links to primary neuroscience sources.

  • Scientific American Ferris Jabr 2012 Accessed: 2026-05-16

    Confirms ~20W brain energy consumption and ~20% of total metabolism.

  • IEA (International Energy Agency) 2025 Accessed: 2026-05-16

    Canonical IEA report. Global data centres: 415 TWh in 2024, projected 945 TWh by 2030. GPT-4 training: ~50 GWh one-time energy cost.

  • Congressional Research Service 2026 Accessed: 2026-05-16

    Official US Congress report. 8 GPUs for 8 hours = 7.92 kW median power draw during training. Source for specific kW figures.

  • PNAS Marcus E. Raichle and Debra A. Gusnard 2002 Accessed: 2026-05-16

    Canonical neuroscience source for the ~20W brain energy figure and 20% of total metabolism.

  • TechCrunch Kyle Wiggers 2025 Accessed: 2026-05-16

    Official figures: 288B active parameters, nearly 2T total, 16 experts, MoE architecture. Confirms Behemoth was in training at the time of announcement.

  • Documents the release delay. Context for the statement about the unreleased model.

  • arXiv / Google Brain Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, Jeff Dean 2017 Accessed: 2026-05-16

    Original paper introducing MoE to neural networks. Describes the mechanism of activating only a subset of parameters per query. Canonical academic source for Mixture-of-Experts.

  • arXiv / Google William Fedus, Barret Zoph, Noam Shazeer 2021 Accessed: 2026-05-16

    First practical implementation of MoE in a language model at trillion-parameter scale. Foundation for LLaMA 4 Behemoth and similar architectures.

  • Canonical source for State-Space Models as a potential transformer successor. Mamba as the leading candidate architecture.

Part 11

The Human Interface

5 Chapters
##

Introduction

1 Topic
Explainers

Human Factors 101 — Cognitive Limits & Biomechanics of Error

10 sources
  • Original paper. Canonical source for the ~7 items in working memory. One of the most cited papers in the history of psychology.

  • PubMed Central Nelson Cowan 2015 Accessed: 2026-05-16

    Retrospective and critical analysis. More recent research suggests ~4 chunks as a more accurate limit. Background for the caveat that 'modern psychologists debated the exact number'.

  • Thinking, Fast and Slow
    Farrar, Straus and Giroux Daniel Kahneman 2011

    ISBN: 978-0-374-27563-1.

    Canonical source for availability heuristic, anchoring, and System 1/2 thinking. The standard popular reference for the heuristics section.

  • Science Amos Tversky, Daniel Kahneman 1974 Accessed: 2026-05-16

    Original academic paper defining availability heuristic and anchoring.

  • Human Error
    Cambridge University Press James Reason 1990

    ISBN: 978-0-521-31419-0.

    Original work introducing the Swiss Cheese Model. Canonical source for the biomechanics of error section.

  • BMJ James Reason 2000 Accessed: 2026-05-16

    Accessible version of the Swiss Cheese Model for a medical audience. Describes it in the context of safety systems.

  • The Challenger Launch Decision
    University of Chicago Press Diane Vaughan 1996

    ISBN: 978-0-226-85175-4.

    Original academic work introducing the term Normalisation of Deviance. Analysis of the Challenger disaster as a case study.

  • The Field Guide to Understanding 'Human Error'
    Taylor & Francis Ltd Sidney Dekker 2006

    ISBN: 978-0-7546-4825-8.

    Accessible industry book on human error in complex systems. Covers normalisation of deviance and the Swiss Cheese Model in an engineering context.

  • Journal of Personality and Social Psychology Baumeister, R. F., Bratslavsky, E., Muraven, M., Tice, D. M. 1998 Accessed: 2026-05-16

    Original paper on ego depletion as the mechanism behind decision fatigue. Canonical academic source for the mechanism (without the specific 200 decisions figure).

  • PNAS Shai Danziger, Jonathan Levav, Liora Avnaim-Pesso 2011 Accessed: 2026-05-16

    The best-known empirical study of decision fatigue. Judges granted parole in ~65% of cases at the start of a session, dropping to ~0% just before a break. An ideal illustrative example.

41

The Human Interface Disasters

6 Topics
General

Google's Simple Homepage

2 sources
  • HuffPost Bianca Bosker 2012 Accessed: 2026-05-16

    Mayer (then VP at Google) described in a Q&A at 92nd Street Y in New York how Sergey Brin built the simplest possible page because he 'didn't have a webmaster and I don't do HTML.' Brin: 'We just kind of stumbled into it'.

    Note: the text suggests both Page and Brin didn't know HTML — Mayer attributes this mainly to Brin. Minor inaccuracy, but it does not change the substance of the story.

  • arnabocean.com (archive) Arnab Gupta 2012 Accessed: 2026-05-16

    Shorter summary with quotes from Brin extracted

Explainers

The Thousand-Dollar Dot: A Personal Burn File

10 sources
Case Studies

The 41-Fold Impossibility — Sold for the Price of a Used Hatchback

4 sources

Citibank $900M 'Fat Finger' Fiasco (2020) — When UX Met Finance

6 sources

The Great Start Button Massacre: A Study in Desktop Hostility

6 sources

Hawaii False Missile Alert

6 sources
42

Bureaucracy Meets Code

4 Topics
Case Studies

Healthcare.gov — The $500 Million 'Error 404'

10 sources

Post Office Horizon — The Algorithm That Made People Thieves

6 sources

India's Aadhaar — The Fingerprint of the Invisible

7 sources

Toeslagenaffaire — The Algorithm That Hunted Parents

9 sources
43

The Human-in-the-Loop Paradox

7 Topics
Explainers

The Automation Paradox — The Skills We Trade for Convenience

6 sources

The Human-in-the-Loop Illusion

5 sources
  • AI and Ethics Luciano Cavalcante Siebert, Maria Luce Lupetti, Evgeni Aizenberg, Niek Beckers, Arkady Zgonnikov, Herman Veluwenkamp, David Abbink, Elisa Giaccardi, Geert-Jan Houben, Catholijn M. Jonker, Jeroen van den Hoven, Deborah Forster & Reginald L. Lagendijk 2022 Accessed: 2026-05-16

    Academic source for the Information/Time/Authority tripartition as conditions for meaningful human control. Formalises the intuition described in the text.

  • Ethics and Information Technology Andreas Matthias 2004 Accessed: 2026-05-16

    Canonical source for the 'responsibility gap' concept: when a human is in the loop only nominally, moral and legal responsibility becomes unassignable. Background for the 'liability sponge' framing.

  • AI Now Institute 2018 Accessed: 2026-05-16

    Describes the structural conditions under which HITL becomes an illusion. Context for the Authority section.

  • YouTube / Don McMillan Don McMillan 2022 Accessed: 2026-05-16

    For those who have made it this far in the bibliography — it had to be here.

    This is not the original source of this sketch, but I couldn't find the right one.

  • YouTube Don McMillan Accessed: 2026-05-16

    Bonus. The author warmly recommends.

    I couldn't link the original clip above so I am including the entire channel.

Case Studies

MiDAS — The $47 Million Shakedown

7 sources

SyRI — The Algorithmic Dragnet of the Underclass

6 sources

Stanislav Petrov — The Manual Override of Armageddon

6 sources

Joshua Brown — The White Trailer Tragedy

5 sources

Boeing 737 MAX — The Algorithmic Stall

6 sources
44

The Illusion of Intelligence

6 Topics
General

The Meta-Fuckup — Hallucinating the Almanac

1 sources
  • adamkorga.com Adam Korga 2026 Accessed: 2026-05-16

    Author's own disclosure page with additional details on AI use in this work.

    The Web Archive hasn't discovered this page (yet), so I can't provide an archival link... but since you're reading this bibliography, that probably isn't a problem.

Case Studies

The Honourable Justice Hallucination Presiding — Mata v. Avianca

5 sources

CNET — The Algorithm That Couldn't Do Third-Grade Math

6 sources

Samsung Electronics — Donating the Crown Jewels

8 sources

The Chatbot Defense — Air Canada and the Hallucinated Discount

5 sources

Vegetative Electron Microscopy — The Ghost in the Machine

9 sources