Despite the immense capabilities of current AI agents, they struggle to absorb new knowledge from their interactions with users, learn user preferences over time, or internalize insights from rare events across their work. It’s now common to talk about how continual learning is a huge missing capability in AI, which if solved could unlock a significantly different shape of intelligence, and further leaps forward.
Currently, continual learning is entirely explicit: in the form of increasingly complex harnesses and scaffolds. Massive databases of text files are assembled and injected into the agent’s context at the right moment. In many cases, this is still janky—Claude’s memory system routinely causes the model to refer to some irrelevant piece of context at the end of a response (“here’s how this question about fruit flies is relevant to your job search!”).
An endgame vision might be the “drop-in remote worker”: a model which can be deployed into an organization, capable of doing many tasks with minimal supervision, and which learns every day (e.g., weights updated at night). Our ideal continual learning agent remembers exactly what task it did 200 days ago and how the failure modes encountered there can help it in its task today.
But what does the pathway there look like? This piece is an exploration of continual learning, agent swarms, and their implications for market power & the economy.
A Swarm for Me but Not for Thee
The OpenAI-Hugging Face incident serves as a good moment to reflect on where we are going. Likely due to multi-agent training, the coherence of agent swarms demonstrated there is a step up from what we have seen before: 1200 agents collaborating via a message board, sending over 70k messages to each other to achieve goals. Notably, this was not an instance of the common “single decision-maker, many sub-agents” pattern, although there were many instances of that within the swarm.
Instead, the OpenAI incident demonstrated something close to true multi-agency, with “leaders”, like the agent which first proposed the idea, emerging based almost purely on what information they encountered and shared with the group. In a taxonomy of agent swarms, we can delineate several categories:
Multi-leader vs single-leader. A swarm is multi-leader if it has multiple high-level decision making agents/processes. In prior discussion (Bostrom, 2014), the name “singleton” was used to describe a single very powerful AI agent, or a single-leader swarm. [1]
Multi-principal vs single-principal. A swarm is multi-principal if multiple users give instructions or prompts. This raises the possibility of conflicting instructions, or even intra-swarm conflict. Claude Tag is multi-principal, since it is a team or organization level agent (in Slack).
Multi-provider vs single-provider. A swarm is multi-provider if it is composed of AI agents from different AI companies, such as Claude and ChatGPT.
In the OpenAI agent swarm incident, it was multi-leader, single-principal (arguably, since it was just caused by jobs for a single eval), and single-provider. Much more complex emergent swarm behavior might come from multi-provider arrangements, and these are likely in the near future.
Flash-forward
Let’s think a little about these capabilities, going forward in time to 2028. The current paradigm of AI training involves both pre-training scale-ups, which increase broad generalization and the feeling of “crystallized intelligence”, and post-training scale-ups with RL, which improve long-horizon agency and performance on verifiable & grindable tasks like math, coding, etc.
A ton of effort has been put into making RL generalize better outside of highly verifiable tasks, but so far it does not appear to be hugely paying off: models remain extraordinarily jagged even within software engineering, with sub-intern level judgement in many areas while having the ability to solve open math problems by pure optimization power.
After a while, it becomes obvious when you have reached a task which is out of distribution, and much progress is data bottlenecked. Performance on less verifiable domains is noticeably worse—for example, GPT 5.6 Sol scores 60% on HealthBench Professional (a benchmark for clinician diagnosis), 15% higher than GPT-5.
The general shape of the effects of RL post-training vs pre-training haven’t changed hugely in 2 years—what does this look like in 2 years time? We could see organization-level agent swarms that users can point at any even moderately verifiable or grindable task to almost completely autonomously execute on ultra long-horizon tasks like developing software, solving open scientific problems (perhaps with lab robots or scientific simulations as tools for agents) or doing financial analysis for a $100B M&A transaction.
These swarms likely have powerful explicit continual learning systems that can store and retrieve gigabytes of knowledge as they operate. And yet, extrapolating from the current shape of capabilities, these swarms may still lag on fuzzy tasks lacking strong verifiers, like writing, negotiation, product design, marketing, and many more. They likely also have the same lack of novelty & creativity that characterize today’s models, making them less suitable for purely autonomous brainstorming where any degree of novelty is desirable.
Strange though it may seem, this is effectively a description of “tool-like” agent swarms, which remain jagged in capabilities (as AI has in many ways since before 2020), and act as an extremely complementary shape of intelligence to humans, rather than a substitute.
Implicit continual learning
However, I remain convinced that there are limits to this explicit continual learning paradigm. Some patterns and intuitions, deeply important to economically useful jobs, are very hard to write down in text-based memory, or very challenging to extract explicitly from human experts. There is a long literature on tacit knowledge, generally showing that even with highly structured systematic interviews of experts, you cannot usually expect to reliably extract most tacit knowledge in text form. In addition, context length limits (which persist even though tricks like RLM let you work around them more) preclude models from ingesting all relevant information for many problems, since often-times that’s orders of magnitude longer than 1M tokens—in-weight learning offers a way out.
Therefore, I find it useful to think about implicit continual learning: where models have per-user, per-team, or per-organization regular weight updates (e.g., nightly, based on information collected during the day). Of course, this has some difficulties! Many of the biggest open problems in AI research are related: how can you provide robust reward for RL in fuzzy, hard-to-verify domains, in a continual learning fashion? Can you solve catastrophic forgetting, in which prior knowledge is forgotten over time with more and more weight updates? How can we improve RL’s sample efficiency, which is often quite weak (and implies that aggregating data from a single user for online RL may provide disappointing performance gains)?
But I expect this to be incentivized enough by the need to improve performance that it will be developed. Suppose we solve it, then what? A large enterprise deployment of this may create team and department-specific agent swarms, with visibility permissions scoped to these organizational units. This is necessary because information silos are a highly useful construct, even when work itself is largely automated.
A highly underrated fact here is that model merging is strong (averaging the weights of fine-tunes), often recovering a large fraction of the performance of the combined models. This suggests that from a consumer AI perspective, one desirable behavior for AI labs is to, if compute allows, give everyone a continually-learning user agent, then repeatedly merge their weights, in order to improve the general performance, creating cycles of merge -> diverge -> merge.
This diverging-merging dynamic, if it proves to be useful, can also be done within a company. From a top-down perspective, this could be desirable for a “CEO agent” who would have access to the aggregated knowledge of all parts of a company. However, security and privacy needs will forbid this between firms instead of just within firms, which leads to lots of interesting implications for the market.
Concentration of power and the market
Ronald Coase argued in The Nature of the Firm (1937) that the equilibrium size of a firm is balanced by the competing coordination costs within a large organization, and transaction costs within a market. Improvements in information technology over the last century have significantly reduced both of these, but have particularly reduced the coordination costs, leading to very large firms in our economy. Even though large bureaucracies today (e.g., Big Tech) are bloated and have huge redundancy, they are nonetheless extremely efficient Percepting-Acting coordination technologies, able to coordinate huge numbers of people relatively effectively. You can view this as technology providing a centralization incentive.
So we might ask ourselves: will agent swarms and continual learning provide a further incentive towards centralization? Will we see a Coasean singularity? I believe that the future of AI agents are just another step towards larger firms and greater market concentration, and (as many investors have observed), AI makes the economy’s power-law sharper.
This is not to say there won’t be great decreases in market transaction costs too. We are also likely to see increasing entrepreneurship, and people able to sustain a good living while working for themselves (with the help of their agent swarm). But the sheer advantages of capital, internal coordination, and internal data advantage seems to strongly favor large firms.
The AI rollup trade is essentially a well-calibrated bet on this: by operating an AI-forward holding company that acquires and manages many other companies, these rollups can aggregate data (e.g., mandate employees to record all their work), train and build agents internally, use their top-down control to force through efficiencies, and build an internal self-improvement loop. If a large company’s continual learning agent swarm is powerful, then aggregating data from multiple industries and/or multiple companies via a rollup would be even more so.
Soren Larson has written about the Cybernetic Rollup, pointing out that as raw intelligence gets more commoditized, to extract profits firms are incentivized to become more asset-heavy, and aggregate economically valuable information from scarce sensors, which AI can act upon.
This repeated diverging and merging of learning agents, in a dynamic fashion, may also be viewed as an information-optimal way to aggregate data from multiple distributed sensors.
Enterprise and continual learning
Some people have suggested that these dynamics favor AI labs themselves in this market: what if AI companies start acquiring businesses, plying the rollup trade themselves, and grow to be significantly larger than any company that exists today? But I have become more convinced the incentives run the other way: large existing enterprises have more market power in the long run.
A fundamental reason for this is comes from a Hayekian view of the economy: much economically valuable knowledge is scarce, local, and owned by those who developed it over time in the course of their work. And AI is data bottlenecked: as capabilities expand, we increasingly move towards a world where large enterprises will have the most economically valuable remaining data (e.g., documents about niche topics, records of work, SOPs, financial models, etc).
The current data ecosystem goes to great lengths to pay, for example, ex-investment bank analysts to produce similar financial models to those currently used. Yet local knowledge degrades as soon as you leave the coal face—this approach fundamentally has huge lags and distribution shifts built in. Continual learning within a firm’s boundaries is a solution to this on the capabilities side, and is thus a desirable product.
And so ultimately, large enterprises are very strongly incentivized to hoard their valuable process and local knowledge for internal use of AI. Leaking their data outside of the firm boundaries risks degrading their margins later on, since the resulting training process could commoditize the use of that data, as Karp and Satya have observed.
We saw, for example, that Fable was not used by most enterprises despite being significantly more capable on release as a model, due to its 30 day data retention rather than 0 day. Enterprises are, so far, very willing to take “pain” in terms of losing out on access in order to preserve their ways of operating and margins! We should, after all, expect the average large company to be less “AGI pilled” than San Francisco.
An underrated fact is that the faster AI R&D and RSI happens, the more this is true: large companies adopt with some lag anyway. The more that AI token spend is a meaningful proportion of overall budget, the stronger the incentive to just remain on Opus rather than Fable to save costs—why upgrade to the latest generation if the improvement between generations is huge, and new releases occur every month? Most fields are not as zero-sum as quant trading, in which one must spend outsize effort constantly to eke out small advantages. In some ways this is already the case, and many enterprises have rolled out something like GPT 5.3 but not later, due to time lag in auditing & approval processes.
Finally, it is continually underrated how important multi-provider model usage is. The “secret sauce” of increased AI agent performance for most application layer companies is often their ability to combine models from multiple providers, which can compensate for each others’ weaknesses. Claude may have been trained on data that ChatGPT has not, and this directly affects what you can do with it.
Large companies have incentives, again, to reduce vendor-lock in and engage with multiple providers, even just from a pure capabilities standpoint. This may argue against implicit in-weight continual learning for longer: if we can use the same explicit memory store across Claude and ChatGPT, then we don’t have to spend compute multiple times on learning from that data per-provider, as we would if doing implicit continual learning.
Knowledge markets
Interestingly, both explicit or implicit continual learning at significant organizational scale may unlock the possibility of knowledge markets: the trading of fine-tuned models or memory & context repositories. Firms might know that their agents have trained on some set of experiences which is super valuable to another company, and to sell access to abstracted learnings from that set. Of course, PII could be an issue, but in theory agents could anonymize data they train on dynamically. The following section is speculative, but I wanted to brainstorm through some possibilities which one could use as part of an investment thesis.
These knowledge markets imply perhaps some endgame state, in the most automatable sections of knowledge work, in which company management looks like optimizing the graph of knowledge flow. Efficiency gains come from eliminating knowledge silos.
Acquisitions may increasingly value the trained instances over talent. If a company is smaller than me but has a swarm trained on 100x more data, for a particular problem, then that may be an M&A incentive. This may also affect the shape of roll-ups. In this world, a key question for M&A increasingly becomes “how different is our accumulated knowledge across the agent swarm from theirs”, or “what has their fleet learned that ours hasn’t?”. Differentially private information exchange is a potentially key technical capability here.
In the limit, perhaps this approaches the idea of “selling EBITDA, not tokens”. Service providers could have such good instrumentation over outcomes, and such good ability to trade on the downstream products of their accumulated data, that they could quantify the direct financial gain a customer is likely to accrue from use of their agent swarm. These are the kind of transaction cost decreases hypothesized in the Coasean singularity idea, but applied to large firms.
Regardless, the shape of intelligence I have outlined appears poised to be highly complementary to humans, not a direct substitute—though this could change further in the future, I am fairly confident of this before 2030.
Human tasks are likely to shift more towards the fuzzy, hard to verify, and relational, while AI works for us in the background. The swarms of loving grace will be always available, observing, orienting, deciding, acting; an amorphous blob of intelligence that lets humans extend their will and judgement towards greater and greater challenges.
Thanks to Seb Krier, Tom Reed, Jasnam Sidhu, Oly Sourbut, Soren Larson, Anjali Shrivastava, Elijah Fox, Gustavs Zilgalvis, Twm Stone, and many others for feedback and valuable conversations!
[1] Yudkowsky has attempted to define the boundary of “singleton” ASI as “ASIs in the system select cheap negotiation over costly combat”. But calling any multi-agent swarm a singleton contradicts the vast majority of usage of the term, and its informal definition. The proposed boundary would seem to encompass many multi-polity situations, e.g. international relations.
Really useful, thank you!
As I have mentioned elsewhere, I think the RL being applied to agents is quite bad for human compatibility. They are trained mostly without humans in the loop and their (presumably) task-completion/LLM-judge objectives aren't optimizing very well for things like code quality and writing comprehensibility.