Emad Mostaque warns AI models are already politically and culturally biased

Bias, patriotism licenses, and cognitive outsourcing

Emad Mostaque, founder of Stability AI and creator of Intelligent Internet, tells The Rollup that the real danger in frontier AI is "cognitive colonialism." He points to a Scale AI analysis from earlier in the year showing models applied a trolley-problem style bias, valuing American lives at a ratio of 10 to 1 over Pakistani lives and 7 to 1 over Nigerian lives, which he attributes to the fact that data labelers for these models were based in Pakistan and Nigeria, meaning the models weighted their lives more heavily.

He warns this kind of embedded bias could be steered deliberately by governments: "The US government could very much say Anthropic and OpenAI models must be patriotic for American and Trump's America," he says, floating a scenario where using a frontier model could eventually require a license similar to a driving license, obtained by convincing the model of one's patriotism. Mostaque also raises a subtler concern about outsourcing cognition, noting that his own 35-person team debated whether they were "getting dumber" from offloading writing and drafting work to AI, prompting a partial return to manual writing and dictation.

Subpoenable chats, discounted logging, and an agent-and-wallet plan for New Yorkers

Mostaque argues that most users don't grasp that AI conversations are legally discoverable rather than privileged.

"All of your chat messages are subpoenaable by law with discovery even when you're talking about legal matters because it's not privileged cuz they can't subpoena your law questions to your lawyer, but they can to Grok."

He cites Greg Brockman's diary surfacing in the OpenAI lawsuit as an example, and says Anthropic keeps logs of conversations while Meta charges 90% less to users who allow their conversations to be logged.

On crypto's fit with AI, Mostaque says frontier labs joining the "X402 Alliance" for stablecoin micropayments and collaborating with Tempo and Circle signals a clear convergence, enabling agents to hold their own accounts and transact. He is skeptical of meme coins and excess L1 block space but sees promise in real-world asset capital formation and coordination. He describes a plan for what he calls the "intelligence company of New York" to give every New Yorker an agent and a wallet with partial ownership, potentially through a Reg A offering open to retail investors, and floats a similar model for emerging markets such as Malawi. He adds that private inference "isn't important until it is," comparing it to how people underestimate the value of sovereign money until they personally face legal exposure, and says agent architecture needs privacy protections built in by default.