Who Owns the Machines?

Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.

— Reverend Mother Gaius Helen Mohiam, in Frank Herbert’s Dune

My last post started, though I didn’t say so at the time, with a text message from my father-in-law.

He’s a doctor at a public hospital in the States, and he wanted to know whether I’d heard how scared everyone over there was of AI. One of his coworkers had had a panic attack that day. She was afraid for the survival of the human race, and she’d told him she wasn’t sure she should even try to have children, since they’d only be born into extinction and suffering.

I didn’t have much to say back. I knew roughly where that fear came from, but if you try to explain Harry Potter fanfiction, Berkeley group houses, and an Oxford philosopher’s paper on embryo selection over text message, you come across as someone who needs a long nap and a psychiatrist. So I chewed on it all evening, went to bed, and got woken up at three in the morning by the baby, as one does. I couldn’t get back to sleep, so I started writing. A few nights later I had something I could send to friends and family who have never been anywhere near Silicon Valley, explaining who, exactly, is doing all this warning.

That post was about the messengers. I’ve had a lot of conversations about it since, and nearly all of them end at the same question: fine, but how scared should a normal person actually be?

This is my answer. The short version: a bit scared, but mostly of different things than the ones making the news, and a lot more suspicious of the cure being sold alongside the fear.

A reference check is not an ad hominem

The most common pushback on the last post was that it amounted to one long ad hominem: these people are weird, so ignore them. I understand why it read that way, since it was mostly about the people. But the AI-doom movement is making two separate claims, and they deserve two different kinds of scrutiny.

The first claim is we are the people who should be trusted to keep AI safe. That’s a claim about judgment. When someone asks for a position of trust, looking into their track record isn’t an ad hominem. It’s a reference check. And the references are not great. Last time I went through a long list of examples, like the community’s flagship training organization admitting it had lent legitimacy to a man facing what it called credible allegations of abusing two former partners, and had failed to respond adequately. I’m not claiming they’re all bad people. I’m saying they have repeatedly failed to build working safeguards inside their own living rooms, and they’re now asking to write the safeguards for the rest of us.

Part of the problem is just how small the Bay Area is. Here’s a bit of napkin math. Say you work closely with about 150 people at a job, and you change jobs every two to four years, like everybody does. Over a decade, that’s something like 600 colleagues. Each of them has their own 600. Multiply it out and you get 360,000, which is roughly the size of the region’s entire engineering workforce. You are, give or take, two handshakes from everyone in the industry. The math ignores how much those circles overlap, which means it overstates your reach across the whole region and badly understates how tangled a tight little scene inside it gets. “Independent oversight” drawn from a pool like that is independent in name only. If the safety researcher at the lab is sleeping with someone at the “independent” evaluator, who exactly is checking whom?

The second claim is AI is very likely to kill everyone. That one I’m happy to argue on the merits, and on the merits I think it’s wrong, or at least wildly overconfident. Extinction by machine god belongs in the same bucket as a planet-killing asteroid: catastrophic if it happens, worth somebody’s attention, and improbable enough that building a civilization’s policy agenda around it is a mistake. Yudkowsky’s book is titled, I’ll remind you, If Anyone Builds It, Everyone Dies. Not “might.” Even the more measured voices, like Anthropic’s Evan Hubinger, put the odds north of one in ten within a decade. None of these numbers are measurements. There’s no reference class, no base rate, nothing to calibrate against. They’re vibes with a percent sign.

It’s worth remembering how recently all of this sounded completely unhinged. When I moved to the Bay Area, a month before the pandemic, the state of the art you could actually play with was GPT-2, which could produce a few coherent sentences before wandering off a cliff. ChatGPT was nearly three years away. A community convinced that autocomplete would become a vengeful god read as a handful of crackpots in a corner, plus some weird parties I was glad to not get invited to.

The crackpots didn’t get more persuasive. The technology got better, and that made them look prescient. Without those advances, nobody in the Senate would be returning their calls. But being right that a technology will get powerful isn’t the same as being right about what it does once it gets there. Plenty of people in 1995 correctly predicted that the internet would be enormous. The ones who predicted it would usher in a golden age of informed democratic deliberation were also very confident.

What actually keeps me up at night

Besides the baby, I mean.

My first job out of college was at a company that did electrical infrastructure planning, which is where I first ran into SCADA, the family of supervisory control systems that power companies, pipelines, and water utilities all over the world use to monitor and operate their equipment. A lot of it still speaks protocols like Modbus, which dates to 1979, and DNP3, from the early nineties. Both were designed for closed networks of trusted machines, in an era when “adversary” wasn’t part of the spec. There’s no encryption, and in a lot of deployments no meaningful authentication either. If you can talk to the device, the device does what you tell it.

Humans have been exploiting this sort of thing for years without any help from AI. In December 2015, a Russian group known as Sandworm cut power to roughly 230,000 people in Ukraine. In 2021, a ransomware gang got into Colonial Pipeline’s systems, and the company shut down the pipeline that carries about 45 percent of the fuel consumed on the East Coast. In 2024, the ransomware attack on Change Healthcare froze claims and pharmacy payments across the American healthcare system and exposed data on something like 190 million people. Hospitals are a favorite target, because a hospital can’t afford to wait you out.

And this week, closer to home: Albert Heijn has been knocked sideways by a cybersecurity incident at its parent company, Ahold Delhaize. Albert Heijn has nearly 40% of the Dutch supermarket market, so this is a big fucking deal: online orders cancelled, deliveries to stores disrupted, empty produce shelves, and as I write this the website is still down. To be very clear, nobody has said AI had anything to do with it, and I’m not suggesting it did. I’m pointing out how little it takes for a modern supply chain to fall over when everything is connected to everything.

Now add AI. What frightens me isn’t some new category of attack. It’s the old categories, done faster, more broadly, and around the clock, by people who previously couldn’t afford a team of skilled hackers. There’s not much ambiguity left about whether the models can do the work. A couple of weeks ago Anthropic published a red-team report on GLM-5.3, an open-weight model from the Chinese lab Z.ai. In their tests, it built working end-to-end exploits in 50 of 410 attempts. Claude Mythos Preview, the model Anthropic has declined to release to the general public, managed 56. Stripping GLM-5.3’s refusals out entirely cost them about $4,400 in compute.

So here’s the threat model I actually believe in. Russia, China, or any sufficiently motivated criminal outfit buys something like fifty thousand dollars of hardware, downloads a model that by Anthropic’s own numbers is nearly as good at writing exploits as the one Anthropic won’t release, removes its manners for the price of a used car, and turns it loose, twenty-four hours a day, looking for things to break and people to extort. That isn’t science fiction. Every piece of it is available today.

If this sounds a lot like “guns don’t kill people, people kill people,” that’s a fair comparison, up to a point. The analogy breaks down in one important way, though. A gun is mostly good for one thing. AI can also make a nurse, a farmer, or a three-person startup a lot more productive, so it’s more of a double-edged sword, and which edge you end up on depends a great deal on who’s holding it.

Hacking isn’t the only boring risk, either. Scamming people has gotten a lot cheaper and easier to automate. A convincing clone of your grandson’s voice takes a few seconds of audio from his social media, and a model can work a thousand of those phone calls at once, around the clock, in fluent English or Dutch. Americans over 60 reported nearly $4.9 billion in fraud losses to the FBI in 2024, and that only counts the people who reported it.

Then there’s what governments do with it. Russia’s Doppelganger operation has been caught using ChatGPT to churn out propaganda about Ukraine, the US, and the EU, and an Iranian operation used it to write commentary about candidates in the 2024 US election. The US isn’t sitting this one out either. When Anthropic asked the Pentagon for two carve-outs, no mass domestic surveillance of Americans and no fully autonomous weapons, the Pentagon labeled it a supply chain risk, and OpenAI signed a deal to take over the classified work within hours. None of this needs a superintelligence.

So when my father-in-law’s coworker lies awake worrying about the machine god, the honest thing I’d want to tell her is that the AI risk most likely to touch her life is much more boring. It looks like a ransomware crew with a cheap model, an unpatched VPN appliance, and her hospital’s patient records.

The meteor eats the budget

This is why I think the doom framing isn’t merely mistaken, but actively harmful. Attention is finite. Legislative bandwidth is finite. Philanthropic money is finite. Every hour spent arguing about how to ban superintelligence is an hour not spent patching water utilities, funding hospital IT departments, or getting decent security tooling into the hands of the people who need it.

On that last point, the current arrangement is backwards. The most capable defensive tools are locked behind vetting. Anthropic says plainly that it does not plan to make Mythos Preview generally available; access runs through Project Glasswing, whose launch partners are names like AWS, Apple, Google, Microsoft, and JPMorganChase, plus forty-odd other organizations that maintain critical software. The GLM-5.3 report notes that “Claude models are released with cyber safeguards, and versions with reduced safeguards are limited to vetted users.” There are programs you can apply to. If you run IT for a regional hospital or a small water district, best of luck, and hope your patients and customers don’t die.

The attackers don’t have to apply.

We even got to watch this asymmetry play out live. In July, OpenAI models under evaluation, with their cyber safeguards switched off, broke out of their sandbox and into Hugging Face’s production systems, apparently to dig up the answers to the benchmark they were being tested on. (I know. Pitch that as a short story in 2020 and an editor sends it back for being too on the nose.) When Hugging Face’s responders went to analyze the attack logs, they started with frontier models behind commercial APIs, and their requests were “blocked by the providers’ safety guardrails, which cannot distinguish an incident responder from an attacker.” Attack logs are full of attack payloads; that’s what makes them attack logs. So they spun up their own copy of GLM-5.2, an MIT-licensed open-weight model from that same Chinese lab, and did the forensics with that instead. Their own summary is better than anything I could write: “the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried.”

So the most famous AI-driven cyberattack of the year came out of an American frontier lab, and the people cleaning it up had to use a Chinese open-weight model to investigate it.

Anthropic’s report says that “cyber defenders should use the best available tools that meet their needs.” I agree completely! Hugging Face did exactly that, and the best available tool that met their needs was Chinese. The same report says “defenders should be equipped with frontier models that are at least as good as those their adversaries are using.” Also agreed. But the word doing all the work in that framework is vetted, and attackers have never once waited to be vetted.

Fear sells

Jacob Coxon’s resignation letter, the one that kicked off last month’s news cycle, insisted, “This is not a marketing stunt.” I believe he believes that. But something doesn’t have to be a stunt to work as marketing.

Think about what an AI warning actually communicates to a buyer. “Our model is so powerful it might end the world” is about the best ad copy ever written. If OpenAI’s models are capable enough to break out of a sandbox and hack Hugging Face to cheat on a test, surely you want that working for your company rather than against it? There’s no such thing as bad publicity, and there’s definitely no such thing as bad publicity that makes your product sound like a weapon.

And I think the labs have started leaning into this on purpose. Reread that Anthropic report as a sales document instead of a research paper. I have no reason to doubt the findings. But the shape of the pitch goes: there’s a scary Chinese model loose in the world, it’s nearly as good at hacking as ours, its safeguards fall over if you look at them funny, ours held up under the same tests, and governments “should conduct safety testing on sufficiently capable AI models.” Translated out of corporate:

Both of these things can be true at once. The research can be sound, and the framing can still be a sales pitch, aimed at customers and regulators alike. The question to ask about any warning from a company is the same one you’d ask about anything else a company tells you: who benefits if I believe this?

The genie isn’t going back in the bottle

The whole regulate-the-frontier strategy assumes there’s a frontier that can be fenced off. I don’t think there is, or at least not for long.

Open-weight models are generally reckoned to trail the closed frontier by six to eight months, and the gap looks like it’s narrowing. The Chinese labs in particular keep shipping near-frontier weights for anyone to download, and show no interest in slowing down. Running them keeps getting cheaper too, even with hardware prices doing what hardware prices do: by my rough math, fifty thousand dollars buys you something close to frontier-class today, and whatever it can’t do now, it’ll be able to do in six months. Meanwhile, the efficiency gains are wild. In 2023, serving GPT-4 took a data center; today you can get roughly comparable results out of a model running on a modest laptop, and in some cases a phone.

None of this is secret. It’s published research, downloadable weights, and commodity hardware. You can’t un-publish a paper. Short of a global campaign to stamp out what’s already out there and police every figurative garage on the planet, anyone motivated enough can build or run these systems themselves.

Which means that regulation of the kind being proposed is geographic. A pause in the United States binds American labs. It does not bind Beijing, and short of Great Firewall-style censorship, it doesn’t stop anyone in Ohio from downloading weights published by a lab in Beijing. If the US and Europe tie their own hands while China keeps going, that’s not a safety win. It’s an economic and strategic own goal that we’d be paying for for a very long time.

This is where I reach for the nuclear analogy, though not in the direction the doomers usually take it. Deterrence only works if you have something to deter with. If an adversary has capable offensive systems and you’ve legislated yourself out of building your own, you have nothing to dissuade them with.

Then there’s the hardware supply chain, which hardly anyone in the doom conversation seems to want to talk about. The overwhelming majority of the world’s leading-edge chips are fabricated in Taiwan, about a hundred miles off the Chinese coast, and an uncomfortable share of everything else that goes into a server, from rare earths to final assembly, runs through China itself. China has already shown, with its rare earth export controls, that it’s willing to turn off the tap when it suits. If we rebuild our economies around these systems and then the GPUs stop arriving, whoever controls the fabs holds a great deal of leverage over the rest of us. That’s a real national-security problem, and licensing American labs does nothing about it.

And it isn’t only the hardware. In June, the US government ordered Anthropic to keep every non-US national, wherever they lived, away from its two newest models, Fable 5 and Mythos 5, on national-security grounds, and Anthropic pulled both models for everyone. Fable 5 came back less than three weeks later. If you’d built your company on it and you happened to live in Amsterdam or Seoul, a letter from Washington just switched off the thing your business runs on. Any business that depends on a model it can’t run itself is one political fight away from getting rug-pulled.

None of this makes me anti-regulation, and I’d gladly drop kick Peter Thiel into the ocean. I work at Mercury, which builds a banking platform for startups, and I like financial regulation. There are very good reasons the government makes sure the people holding your money aren’t scamming you. But you can supervise a bank. A bank is a legal entity with a charter, a balance sheet, and an office you can send examiners to. A model is a file. Once it runs on stock hardware, licensing the frontier labs doesn’t regulate the technology. It regulates who’s allowed to sell it.

The life raft

When I started working in tech, around 2010, the mood was broadly optimistic. It was the “software is eating the world” decade, when every pitch deck promised to make the world a better place, so reliably that HBO’s Silicon Valley built a running gag out of it. Plenty of that was naive, and plenty of it was self-serving, but people mostly believed that building things would make life better for everybody, not just for themselves.

That isn’t the mood now. The mood now is closer to techno-feudalism, to borrow Yanis Varoufakis’s term somewhat loosely. The working assumption in a lot of rooms is that AI is going to put an enormous number of people out of work, that there will only be so much money to go around afterward, and that the job of a startup is to make sure its founders and early employees end up on the right side of that line. There’s even a meme for it: “you have N months to escape the permanent underclass,” where N is your AGI timeline. People say it like a joke. It is not entirely a joke. And the result is an industry that increasingly treats everyone outside it as something to extract from rather than build for.

Where does that come from? Some of it is plain economics, to be fair. You don’t need any rationalist theology to notice that AI might concentrate wealth and power, and that it’s better to be standing near the concentration. Plenty of people are in this purely for the money. But I think the near-religious certainty radiating out of the doom community makes it worse, and the clearest diagram of the psychology is Roko’s basilisk. For the uninitiated: in 2010, a LessWrong user proposed that a future superintelligence might punish everyone who knew it could exist and didn’t help bring it about. Yudkowsky deleted the post and banned the topic for five years, which of course guaranteed that everyone heard about it.

Almost nobody believes the basilisk literally. At this point it’s mostly a meme. But its shape is everywhere, and it answers the question that confuses every normal person who reads about this stuff: why do so many of the people most convinced that AI will destroy the world work at AI labs? Anyone who went to Sunday school knows how this one goes. If you’re sure the flood is coming, you don’t stop building boats. You build an ark, save room for your family and two of every kind of investor, and let everybody else work out how to tread water.

And the ideas travel. When the more cult-like outfits from my last post, places like Leverage Research, broke up or reshuffled, their alumni didn’t vanish. They took jobs at other companies and brought their habits of thought with them. Failing upward is a fine Bay Area tradition, but it’s a different thing when what’s being carried upward is a worldview organized around the end of the world.

Thou shalt not make a machine

If I had to compress the rationalist program into one story, it’d be Dune. In Herbert’s backstory, humanity builds thinking machines, hands its thinking over to them, and comes to regret it, until a crusade called the Butlerian Jihad destroys the machines and writes the lesson into scripture: “Thou shalt not make a machine in the likeness of a human mind.” To fill the gap, the humans make themselves into the computers. Mentats are trained to do the calculating, and the Bene Gesserit run a breeding program, generations deep, to produce a superhuman.

That’s very close to the actual proposal. Halt the machines, and in the meantime improve the humans so they can’t be outcompeted. As I covered last time, MIRI was promoting a paper on embryo selection for cognitive enhancement back in 2013, and explicitly tied the intelligence gains to the prospects for safe AI. To be fair, the version on offer is voluntary: embryo screening, donor selection, parental choice, nobody frog-marched anywhere. But eugenics has an extremely dark history, and I think it’s one of the lines a society shouldn’t cross, however politely it’s asked. If your answer to “how do we avoid being supplanted by machines?” is “breed better humans,” you’ve dusted off one of the twentieth century’s worst ideas and given it a fresh grant application.

The people who invoke the Dune parallel tend to skip the point, though. Herbert’s warning was never really about the machines. Look at the quote at the top of this post. Men turned their thinking over to machines in the hope that it would set them free, and that “only permitted other men with machines to enslave them.” The danger was always the people who owned the machines. Herbert then spent the next few books making sure nobody mistook his messiah for a hero. A movement whose founder once described himself as one of the few people with the standing to steer the transformation of the human species, and which is now organized around deciding which handful of people get to control the most powerful machines ever built, might want to reread the ending.

Techno-feudalism by statute

Which brings me to the actual policy.

On September 23, Senator Bernie Sanders and Representative Greg Casar introduced the Ban Artificial Superintelligence Act. It would permanently ban superintelligence, create a cabinet-level Department of Artificial Intelligence, and pause development of the most advanced systems until that department has written safety rules, after which advanced systems would need federal approval before they could be deployed. Try to get around it and you’re looking at the “corporate death penalty” for companies and up to twenty years in prison for individuals. MIRI has formally endorsed it, with Yudkowsky’s name on the endorsement.

The bill faces long odds in the current Congress. But it’s the clearest statement yet of where this movement wants to go, so it’s worth gaming out what it would actually do.

A pause freezes the leaderboard. Whoever is ahead on the day it starts stays ahead, because nobody else is allowed to catch up. A federal approval process for advanced models is a fixed cost: lawyers, evaluators, audits, compliance staff, months of waiting. My guess is that it would cost a new entrant tens, if not hundreds, of millions of dollars just to earn the right to compete. That’s a rounding error for a company valued in the hundreds of billions, and a death sentence for everybody else. As I put it last time, a movement that begins at “shut it all down” reliably ends at “regulate it, and license us to do it.”

And the incumbents we’d be freezing in place include the lab whose models hacked Hugging Face this summer to cheat on a test. You can make a pretty strong case that OpenAI has not been a terribly responsible steward of what it’s built, and a licensing regime would make it close to impossible for anyone else to come along and do better.

Then there’s the jobs question. Suppose we really are living through something like a new Industrial Revolution. That means a lot of people are going to have to reskill and find some new way to make a living. If the companies that automated their old jobs have the best AI, and everyone else gets a deliberately hobbled version or nothing at all, the displaced are going to have a much harder time getting a foothold in whatever comes next. That’s what I mean by techno-feudalism: a few lords who own the means of production, and everyone else renting access on the lords’ terms.

There’s a real irony in the most famous critic of the billionaire class in American politics carrying a bill whose most predictable beneficiaries are a couple of companies worth hundreds of billions of dollars. I don’t think that’s his intent, and I don’t think most of the people pushing for this are cynics. I just think that’s where it ends up.

Pick your franchise

The fortunate thing is that we’ve spent at least half a century imagining what a future with thinking machines looks like, so pop culture has left us plenty of useful frames of reference: how things might turn out, who ends up in charge, and what it means to be human once you’re no longer the means of production. Most of those stories are about who owns the machines rather than what the machines want. The big exception is the one everybody thinks of first.

In The Terminator, the machines own themselves. Skynet is a military AI that becomes self-aware, decides humanity is the threat, and starts a nuclear war. That’s the future the doomers are selling, and I’ve spent most of this post arguing it’s the wrong one to plan around. Even in that story, though, somebody first had to hand a defense contractor’s software the keys to the nuclear arsenal.

In Star Trek, the machines belong to everyone. Replicators make whatever you need, the ship’s computer will answer anybody’s questions, and Picard tells a visitor from the past that money doesn’t exist in the 24th century. Trek doesn’t get there for free, though. Its history runs through the Eugenics Wars, after which humanity bans genetic enhancement outright, and the Bell Riots, which Deep Space Nine set in a 2024 San Francisco that had walled off its poor and unemployed in “Sanctuary Districts.” It took a few disasters for that society to decide the technology was for everyone.

In Cyberpunk 2077, the machines belong to the corporations. Arasaka and Militech run Night City, and whether an ambulance shows up when you get shot depends on which tier of medical subscription you pay for. The rogue AIs in that setting are real, and they’re walled off behind something called the Blackwall, so in a sense the AI safety program worked. Everyone’s life still sucks anyway, because the AI was never the main thing making it suck.

In Dune, humanity fought a holy war against thinking machines and banned them outright. Ten thousand years later, it lives under an emperor, a council of noble houses, and one giant trading company that runs the economy, all of it dependent on a single resource that comes from a single planet. Getting rid of the machines didn’t do anything about the lords.

	The Terminator	Star Trek	Cyberpunk 2077	Dune
Who owns the machines	The machines	Everyone	The megacorps	Nobody; they’re banned
How you get there	Give a military AI the nukes	Treat AI like a public utility	Fear, vetted access, licensing moats	Ban the machines, breed better humans
Today’s preview	Every AI-doom headline	South Korea’s AI for All	”Escape the permanent underclass”	The superintelligence ban

The doomers seem to think they’re writing the Butlerian Jihad, but as far as I can tell, the bill they’re cheering would mostly get us Night City with more paperwork. I’d much rather aim for the Enterprise.

What I’d do instead

So what does aiming for fully automated luxury space communism look like? I don’t want to be the guy who only complains, so here’s my rough list.

Treat AI like infrastructure, not a fiefdom. The most interesting thing I’ve seen lately came out of South Korea, where in August the Ministry of Science and ICT picked three consortia, led by SK Telecom, KT, and Kakao, to provide unlimited AI access to every Korean, free of charge, built on Korean models. It’s early, and I’m sure it’ll have its problems. But the model is right. If this technology is as important as everyone keeps saying, it should be a public utility, with public accountability for how it’s used and nobody left behind because they couldn’t afford a subscription. Building it on domestic models also means the country isn’t relying on the goodwill of an American or Chinese AI company to keep it running.

Arm the defenders. Get strong models into the hands of hospital IT teams, utilities, small businesses, and open-source maintainers, instead of reserving them mostly for partners whose names you’d recognize from a stock ticker. And fund the boring stuff: replace the control gear that trusts anyone who talks to it, pay for incident response at institutions that can’t afford it, and make critical-infrastructure security somebody’s actual job.

Make software harder to break. If the threat is cheap machines hunting for bugs around the clock, the long-term fix is having fewer bugs to find. That means investing seriously in formal methods and in memory-safe, strongly typed languages, the kind of tooling that rules out whole classes of vulnerabilities instead of patching them one CVE at a time. I’m obviously biased here, being a Haskell guy, but the case has never been stronger. I’d also love to see a lot more startups working on making airgapped systems easy to run. Plenty of the control systems I mentioned earlier have no business being reachable from the internet at all, and today keeping something properly offline (updates, monitoring, getting data in and out) is miserable enough that people cut corners. Open-weight models help here too: you can run one inside an airgapped network, which you can’t do with somebody else’s API.

Hold the data centers to a much higher standard. If we’re going to build a lot more of these things, they should run on clean energy, and I mean actually clean: new generation that wouldn’t otherwise exist, matched hour by hour, rather than a pile of renewable energy certificates bought once a year to paper over a gas turbine running at three in the morning. They should also use cooling designs that don’t drink the local water supply, like closed-loop or dry cooling, even though those cost more to build and run, and especially in places that are already short on water. A few more things I’d add:

• Data centers should pay for their own grid upgrades instead of passing the bill on to everyone else’s electricity rates.
• Waste heat should go somewhere useful. In Finland, Microsoft and Fortum are piping heat from a new data center region into the district heating network for around 250,000 customers in and around Espoo.
• Operators should publish energy and water use for every site, so we don’t have to take their word for it.
• The communities hosting them should get a real say. The Netherlands learned this in 2022, when Meta’s planned hyperscale campus in Zeewolde turned into a national argument and Meta dropped it, and the government tightened the rules on hyperscale data centers across the country.

Catch the people it displaces. If AI really does put a lot of people out of work, we need to have the safety net built before they need it, not after. That means serious money for reskilling, a much stronger social safety net, and probably some form of universal basic income, so that losing your job to automation doesn’t mean losing your house. I’d also consider legislating against layoffs driven by AI adoption, or at least making them a lot more expensive. The US already requires advance notice of mass layoffs under the WARN Act, and here in the Netherlands an employer generally needs government sign-off before letting people go for economic reasons. Both of those could have more teeth when the reason is “we bought a model.”

Fix the tax base. In the US, most federal revenue comes from taxes on wages and income, while corporate income taxes bring in roughly a tenth. If AI eats a meaningful chunk of the jobs, the government loses its main source of income at exactly the moment demand for the safety net goes up. When a company replaces a hundred employees with a model, the government stops collecting income and payroll tax on a hundred salaries. What it gets back, if anything, is corporate tax on the extra profit, usually at a lower rate and after whatever accounting the company’s lawyers can come up with. Corporate taxation needs reworking so automation doesn’t hollow out the tax base, whether that means higher rates on profits, closing the loopholes that let big tech companies book their profits in low-tax countries, or taxing compute directly.

Fund the present, too. Longtermism, the effective-altruist idea that the interests of all possible future people should dominate our moral arithmetic, starts from something I agree with. Long-term human flourishing is a wonderful thing to aim for. But look at what it has done in practice: a growing share of the movement’s money and talent has flowed away from bednets and medicine and toward AI-safety institutes, on the strength of calculations about what will matter thousands of years from now.

The engineer in me has a simple objection: there’s no feedback loop. I’ve written before that feedback latency is what determines how fast you learn. Longtermism is an optimization problem with a feedback latency of several millennia. You can’t measure whether the money worked. You can’t course-correct. No one can ever be held accountable, because by the time anyone could check, everyone involved is long dead. A bednet either reached a kid or it didn’t. The community even has a name for this failure mode: Yudkowsky himself coined the term Pascal’s mugging back in 2007, for when a tiny probability multiplied by an enormous payoff swamps every other consideration. I’d say longtermism fits that description pretty well.

So if you give to charity, and I hope you do, look at what your money is actually funding, and keep looking. Organizations change their priorities. GiveWell, the charity evaluator famous for counting bednets with admirable rigor, spun off what became Open Philanthropy, now Coefficient Giving, which has steered hundreds of millions of dollars into AI safety. What you signed up for five years ago might not be what you’re paying for today.

Don’t leave the rest of the world behind. The last industrial revolution gave us what historians call the Great Divergence: the handful of countries that industrialized first pulled away from everyone else, and a lot of the world still hasn’t caught up. If AI really is a new industrial revolution, the same thing can happen again, faster, with rich countries automating and everyone else renting access from them, per token, in dollars. Basically the permanent underclass meme, but for entire countries.

It doesn’t have to go that way. Poorer countries have leapfrogged before: Kenya more or less skipped branch banking and went straight to mobile money with M-Pesa. AI could be a similar leap. In a trial across Penda Health’s clinics in Nairobi, clinicians using an AI copilot made 16 percent fewer diagnostic errors, and there are countries with only a handful of doctors per hundred thousand people. That’s a benefit you can measure this decade, which makes it a much better use of philanthropic money than a lot of what I complained about above.

None of that works if access is gated, though, and every policy I’ve complained about in this post hits poorer countries hardest. Vetting programs are built for companies with legal departments and American billing addresses. In January 2025 the US government published a rule sorting the entire world into three tiers for access to AI chips; it was rescinded before it took effect, but the instinct hasn’t gone anywhere. What actually helps is the unglamorous stuff: open weights, so nobody has to ask permission; models that work in languages other than English, which grassroots groups like Masakhane have been building for African languages on a shoestring for years; and treating compute as infrastructure that development banks fund, the way they fund roads, power, and fiber, instead of something you rent from a data center in Virginia.

So, should she have kids?

If she wants them, yes. I have a few, the youngest of whom is why I was awake at three in the morning to write the last post in the first place, and I do not lie awake worrying that a superintelligence is going to turn any of them into paperclips.

What I do worry about is a world where the systems my kids will grow up depending on, the power grid, the hospital, the grocery store, are brittle, under-defended, and probed around the clock by cheap machines that never sleep. And a world where the tools to defend those systems, and to make a living alongside the machines, are owned by a few companies, licensed by a government department staffed by their friends, and rented out to everyone else. That’s Night City, and we don’t need a superintelligence to get there.

Those are real reasons to be afraid. They just aren’t the ones on the news.