Agents must not run on data exhaust
LLMs and agents have dramatically expanded the scope of what we now call “data” — and, in doing so, have done something stranger to the data value chain than most people have noticed: they have bent it into a loop. A loop is a powerful thing. It is what lets a system learn and improve over time. It is also what lets a system collapse, feeding on its own bad output until the quality turns to nonsense. The same loop runs both ways. One of the key things that decides which way it runs is whether the data flowing through it was created on purpose, or merely scavenged from whatever happened to be lying around.
.
That is the conviction this whole Substack will start from: agents must not run on data exhaust — and far too many of them do. We were told for years that data is the new oil, something that pools beneath the organisation waiting to be pumped and refined. It was a poor metaphor then; in the age of agents it is a dangerous one, because it tells you the valuable data is already there for the taking. It almost never is.
It has been a long time since I last wrote here, and AI has reshaped the data landscape in the gap — exhilarating, and in equal measure hard to keep up with, let alone make sense of. This piece is my way back in: why, in a world increasingly drowning in machine-made data, deliberately creating data worth trusting matters more than it ever has. Let’s start with how much the word “data” now has to carry.
What we now call data
The category “data” has exploded.
For most of the history of our industry, “data” meant the structured, tabular numbers that fit in rows and columns, typed and schema’d, the kind of thing you could put in front of a BI tool. Text, images, audio and video were “data” only in the loosest sense. You could store them (in an object store, not a database), but you could not do much with them.
That boundary has gone. “Data” now encompasses text, images, audio and video; it can also refer to agent outputs, tool calls, reasoning traces, application code, webcam footage and the synthetic data the models generate themselves. The deeper shift is in where the boundary sits. Data used to be bounded by what fits in a table, because that was a pre-requisite to derive insight from it. (That was true whether you were doing analytics or most flavours of machine learning.) With the birth of language models, that is no longer true: it is now bounded by what a large language model can reason over, which today is pretty much everything we can represent digitally.
This is thrilling: it opens up whole new problem types that computers simply could not touch before.
How you impose order on that data sprawl is a really hard question. I am going to leave it hanging for now, because it is the subject of the next piece. The more urgent question, for now, is who, or what, is now consuming all of this?
Agents: a strange new data consumer
For most of the history of our industry there were only ever four ways to drive value from data, and every one of them ran on the same thing: a table. Descriptive: aggregate the rows to work out what happened — sum a column, count the records, group by a dimension. Diagnostic: dig into why, slicing that same table by dimension and comparing a metric across cohorts, or reading the correlations between its columns. Predictive: lay the data out as a feature matrix — one row per example, one column per signal, plus a column for the thing you are trying to predict — and fit a model to it. Prescriptive: hand the decision to the data itself, a pricing model or an approval engine that will not fire until it is handed a row of typed, expected inputs. Four levels of ambition, one substrate beneath all of them: always rows and columns.
Now hand an agent data in the broad sense, and what it can do looks nothing like that list: summarise a thousand documents, riff on a theme or brainstorm against one, build an argument or take one apart, propose a hypothesis worth testing, decompose a problem into constituent parts, turn a messy situation into a diagram or a video. None of this is analytics. It is a different kind of work entirely, and the material it runs on is no longer confined to rows and columns.
In the old world, structure was the price of admission. Dashboards, recommendation engines, decisioning apps: rigid programs that would not start without typed, schema’d inputs. The one apparent exception that proves the rule: a narrow class of machine learning - e.g. sentiment analysis - ingested raw text. But it delivered a structured output, a score or a label, and slotted straight back into the structured world. Unstructured data was never reasoned over in place. It was something you converted into structure as fast as possible, so the real tools could use it.
Meet the large language model, then, and meet it for now through a single face: as a data consumer. It is the first widely deployed thing that consumes unstructured data as unstructured data, reasoning over it directly with no conversion-to-structure step in between. Its appetite, the context window, is finite, a hard token budget, but it is structure-agnostic. It will take almost anything, with none of the schema constraints we built the entire discipline around.
It is tempting to read that as liberation. Structure is dead; just hand the model the pile. It is not. Removing the requirement for structure does not remove its value. The context window does not care whether you hand it a pristine star schema or a heap of raw transcripts, and that is exactly the problem. The model will accept anything, but it reasons well only on faithful, well-organised context. Structure stops being a technical requirement and becomes a quality choice.
The machine that is also the material
Now I can put back the two faces of the model I asked you earlier to ignore.
Let’s categorise everything that touches data by what it does to the data. Some things create it, producing data that represents the world in some way: trackers, sensors, transactions. Some process it, turning data into more data: a SQL query, a dbt model, a training run. And some consume it, turning data into a decision, an action or an insight. These three roles were always kept apart, each played by its own kind of component in a production line. The large language model is the first technology that can play all three roles:
it is a source of data, generating text, code, decisions and traces, and driving an explosion in how much data the world produces;
it is a processing step, a station on the line that transforms its inputs into something new; and
it is a consumer, reasoning over the context window to reach a decision and take an action.
This is unprecedented on two counts. We have never had a single component that is at once the raw-material supplier, the machine and the customer. And it occupies all three roles having shed the structure requirement that defined every one of them before. A line that used to run in one direction, through distinct stations, collapses into a single structure-agnostic component, and that bends the manufacturing line into a loop. The machine is now also the material.
This is the idea the rest of this Substack will keep returning to. Everything that follows, the promise and the danger both, comes from those three roles collapsing into one.
Choking on our own exhaust
The risk falls straight out of the loop, which is why it needs no separate motivation. If the line loops, and we keep feeding it the exhaust it throws off, quality does not merely stall. It compounds downward. The assembly line starts eating its own output.
This is not a thought experiment. It is model collapse: the finding, published in Nature, that models trained recursively on their own generated data degrade across generations until their output turns to nonsense. It is context rot: the documented result that frontier models get worse as you cram more into the context window, not better. It is agents grounded on the undifferentiated output of other agents. At machine scale, “don’t build systems out of a byproduct” stops being good hygiene and becomes something closer to existential.
A caveat, because this is exactly where the careful reader objects: none of this is an argument against synthetic data. Model collapse is what happens when a model is trained recursively on its own output with no fresh, real signal coming in — not whenever a machine has a hand in making the data. Synthetic data created on purpose, to fill a known gap or stress a known edge case, is some of the most useful data there is. The line that matters is not human-versus-machine origin. It is whether the data was made for a purpose, by design, or scavenged as the undifferentiated byproduct of something else.
But the same loop runs the other way, and this is the hopeful part. Feed the loop faithful data, made to spec, and quality compounds upward. That is what a learning system actually is: manufacture a decision, deliberately create data about it (what was decided, how, and the outcome it drove), reflect on whether that outcome was the one you wanted, and feed that back so the next decision is better. Over time, reliably, it improves.
Collapse and learning are the same loop. The only difference is whether the data flowing back was created or scavenged. That single choice decides whether quality compounds up or down, and it is more or less the whole argument in one sentence.
The lovely example of data that we have identified agents need, and need to be manufactured to spec, is the decision trace — a phrase I am borrowing from Jaya Gupta and Ashu Garg of Foundation Capital, who named it late last year. A decision trace is a record of how a decision was actually made: what was weighed, what was discarded, why one path was taken over another, who signed it off. Almost nothing emits this: it needs to be deliberately created. It survives, if at all, as fragments buried in call transcripts and meeting minutes, never with the granularity, fidelity or provenance an agent would need to learn from it. And it is exactly what an agent that makes decisions needs: a faithful account of how humans reason through a choice, to follow or to disagree with.
Notice what that means. Even now, with machines generating more data than we know what to do with, some of the most valuable datasets still do not exist. They have to be manufactured on purpose, because no system throws them off as a byproduct.
When data is effectively infinite and machines make most of it, the scarce thing is no longer data. It is data that is actually useful. The data that is worth trusting. The abundance of exhaust does not make deliberate creation less valuable; it makes it close to the only thing that matters.
What I want to write about
That is why I am writing again: to think in public about how we create valuable data worth trusting, in a world where that has never been easier to fake or more expensive to get wrong. Data Creation - the thing this Substack is named for - is just that: the deliberate act of making data for a purpose, not the scavenging of data made for some other one. Some of it will be conceptual. A lot of it will be the unglamorous, specific things we are learning from doing the work. (Snowplow is where most of that learning comes from, so it will show up here, as evidence rather than a sales pitch.)
The next piece picks up exactly where this one stops: with the layer the whole industry is currently arguing about, and the one term it keeps getting confused with. Everyone has started talking about a “context layer.” Almost no one agrees what it is, and a good many people are quietly assuming it is just a semantic layer with an agent bolted on top. It is not, and the difference, it turns out, matters enormously.
More soon.