As I have mentioned elsewhere, I think the RL being applied to agents is quite bad for human compatibility. They are trained mostly without humans in the loop and their (presumably) task-completion/LLM-judge objectives aren't optimizing very well for things like code quality and writing comprehensibility.
I think (a) poor quality code is an issue for agents roughly as much as it is for humans, so doesn't come apart so much from human-written code. readability may be, but so far AI has been incentivized to make code readable at least partly because it scales well (process reward/LLM judge rubrics use models which are trained on readable code, and you want to stay as in distribution as possible). Another reason why AI code is readable is simply because AI is made by SWEs who have a particular culture—culture is strongly underrated in general here (b) my point about human-compatibility was at a higher level of abstraction: even if the code is all in Assembly that wouldn't affect my point, because the human would be there to do product design/other hard to verify tasks involved in the production process. Given that I'm just talking about the next few years here, however far you extrapolate capabilities in that timeframe I should always be able to credibly claim there will be some human in the loop involved in hard to verify tasks, and thus the logic holds.
> poor quality code is an issue for agents roughly as much as it is for humans, so doesn't come apart so much from human-written code.
GPT-5.6 Sol is much more stubborn than me and will hillclimb tiny improvements rather than rethink its design. I assume this is because it doesn't have a holistic view of codebases but can write lots of code.
I assume the LLM judge models also care more about local properties than overall code structure.
> even if the code is all in Assembly that wouldn't affect my point, because the human would be there to do product design/other hard to verify tasks involved in the production process.
If the AIs are much better at coordinating amongst themselves than with humans, they will do more things "internally" even if they're worse at it, or do more expensive workarounds to avoid humans in the loop (e.g. for product design, where a skilled human might be able to see the correct shape of a product immediately, stubborn/spiky LLMs might be able to brute-force it with large-scale A/B testing and analytics).
> GPT-5.6 Sol is much more stubborn than me and will hillclimb tiny improvements rather than rethink its design.
Yes, but I see this as a pure capabilities issue: GPT-6 won't be able to improve at the production of more complex software unless it can do more "rethinking its design".
> If the AIs are much better at coordinating amongst themselves than with humans, they will do more things "internally" even if they're worse at it
In theory yes, but if that's the default behavior I think it would not be the revenue-maximizing move as long as "if they're worse at it" is true. If the choice is between less human in the loop but the AI has to do deployment + user A/B tests, vs more human in the loop but the human does product design, the economically efficient version should be the latter. Human labor is just not that costly and the more expensive AI workaround using A/B testing imposes a time bottleneck.
> Yes, but I see this as a pure capabilities issue: GPT-6 won't be able to improve at the production of more complex software unless it can do more "rethinking its design".
Software design as humans do it is a response to particular human limitations; LLMs don't have the same ones. In this case, yes, doing more humanlike software design would probably improve its performance, but the shortest and fastest path to better performance probably isn't being more like humans, but something weirder which works better for GPT-6.
> Human labor is just not that costly and the more expensive AI workaround using A/B testing imposes a time bottleneck.
An hour of skilled human labour is more costly than, at this point, tens to hundreds of millions of tokens from a frontier AI, which is enough to build several modestly complex apps from scratch. The time bottleneck should be partly background-able in this case, and it should be favourable if the AIs can narrow down the search space enough themselves or the thing being designed is popular enough to get feedback very fast.
There are other (somewhat questionable, but not impossible with enough "narrow intelligence") possible workarounds for lack of human input/lack of proprietary datasets, in my opinion, since there's lots of data out there, or reasonably easy to acquire, awaiting clever interpretation. Consider for example demographically weighted LLM sampling as a survey alternative (https://arxiv.org/abs/2209.06899) and things like hedge fund "alternative data" (satellites etc). I suppose this could resolve itself as all data sources becoming costlier, but the marginal cost of handing out data to an extra organization is zero.
Also, the human has to be cheaper than lots of AI time even after the possibly-substantial costs of integrating them into the AI workflows.
Things here were that were new to me and changed my thinknig:
-the model-merging point
-the idea of updating the weights rarely, like once per day. (kind of obvious since ML also does gradient descent only once per batch - makes sense that it would start like that, using a lot data for each step, and not like humans that update constantly to some extent).
do you have any view on how hard it will be to make implicity continual learning happen? cf Dwarkesh's point that sample efficiency seems to have not improved that much at all.
Gotcha. My current vague understanding is that to get genuinely new capability out of RL, you need thousands of *sequential* steps (while keeping tasks in a learnable difficulty range). So it does seem like it would need to get a lot more sample efficient to be commercially viable for learning tacit knowledge even in a once-per-day setup, and even moreso for learning day-to-day about new clients, new situations, etc. (not necessarily saying you disagree, just writing it out for myself)
The conversation around agent swarms keeps mistaking text harnesses for permanent architecture. Pumping context windows with megabytes of scraped operational logs is not learning. It is an expensive, lossy projection. Context compaction permanently erodes causal conditions, and multi-agent transcript overhead burns margins before an action ever executes. 💡
Tacit knowledge resists token serialization. The real competitive moat lives in the Parametric Coase Invariant. When an organization's agents learn, that adaptation cannot survive as external markdown files without collapsing under context drift. It must settle directly into the weights through localized, orthogonal null-space updates. 🧠
Think about what happens when you run cyclical weight merges across department-level fine-tunes. If your update vectors project exclusively into the null space of foundational competencies, catastrophic forgetting drops to zero. You do not need to ship raw proprietary telemetry to an external foundation lab. You keep the entire gradient delta on local silicon. 🛡️
This completely flips Ronald Coase's 1937 calculus. Internal coordination friction plunges toward zero because thousands of specialized sub-agents coordinate over deterministic state lattices rather than conversational chat interfaces. Market transaction costs stay stubbornly high because external vendors cannot inspect or verify un-attested agent traces without leaking their own operational alpha. 🏢
Large enterprises will not be disintermediated by foundation model labs. The labs sell commoditized compute commodities. Enterprises own the scarce, physical ground truth. When cybernetic rollups assemble distributed fleets of local agents and merge their daily parametric deltas on private clusters, they are not buying tokens. They are accumulating private mathematical representations of physical business operations. ⚙️
If the boundary of the firm is now dictated by the fidelity of your in-weight parametric fusion rather than your context window budget, how does your runtime prove that a merged agent weight preserves foundational invariants before dispatching into live production? 🌐
As I have mentioned elsewhere, I think the RL being applied to agents is quite bad for human compatibility. They are trained mostly without humans in the loop and their (presumably) task-completion/LLM-judge objectives aren't optimizing very well for things like code quality and writing comprehensibility.
I think (a) poor quality code is an issue for agents roughly as much as it is for humans, so doesn't come apart so much from human-written code. readability may be, but so far AI has been incentivized to make code readable at least partly because it scales well (process reward/LLM judge rubrics use models which are trained on readable code, and you want to stay as in distribution as possible). Another reason why AI code is readable is simply because AI is made by SWEs who have a particular culture—culture is strongly underrated in general here (b) my point about human-compatibility was at a higher level of abstraction: even if the code is all in Assembly that wouldn't affect my point, because the human would be there to do product design/other hard to verify tasks involved in the production process. Given that I'm just talking about the next few years here, however far you extrapolate capabilities in that timeframe I should always be able to credibly claim there will be some human in the loop involved in hard to verify tasks, and thus the logic holds.
> poor quality code is an issue for agents roughly as much as it is for humans, so doesn't come apart so much from human-written code.
GPT-5.6 Sol is much more stubborn than me and will hillclimb tiny improvements rather than rethink its design. I assume this is because it doesn't have a holistic view of codebases but can write lots of code.
I assume the LLM judge models also care more about local properties than overall code structure.
> even if the code is all in Assembly that wouldn't affect my point, because the human would be there to do product design/other hard to verify tasks involved in the production process.
If the AIs are much better at coordinating amongst themselves than with humans, they will do more things "internally" even if they're worse at it, or do more expensive workarounds to avoid humans in the loop (e.g. for product design, where a skilled human might be able to see the correct shape of a product immediately, stubborn/spiky LLMs might be able to brute-force it with large-scale A/B testing and analytics).
> GPT-5.6 Sol is much more stubborn than me and will hillclimb tiny improvements rather than rethink its design.
Yes, but I see this as a pure capabilities issue: GPT-6 won't be able to improve at the production of more complex software unless it can do more "rethinking its design".
> If the AIs are much better at coordinating amongst themselves than with humans, they will do more things "internally" even if they're worse at it
In theory yes, but if that's the default behavior I think it would not be the revenue-maximizing move as long as "if they're worse at it" is true. If the choice is between less human in the loop but the AI has to do deployment + user A/B tests, vs more human in the loop but the human does product design, the economically efficient version should be the latter. Human labor is just not that costly and the more expensive AI workaround using A/B testing imposes a time bottleneck.
> Yes, but I see this as a pure capabilities issue: GPT-6 won't be able to improve at the production of more complex software unless it can do more "rethinking its design".
Software design as humans do it is a response to particular human limitations; LLMs don't have the same ones. In this case, yes, doing more humanlike software design would probably improve its performance, but the shortest and fastest path to better performance probably isn't being more like humans, but something weirder which works better for GPT-6.
> Human labor is just not that costly and the more expensive AI workaround using A/B testing imposes a time bottleneck.
An hour of skilled human labour is more costly than, at this point, tens to hundreds of millions of tokens from a frontier AI, which is enough to build several modestly complex apps from scratch. The time bottleneck should be partly background-able in this case, and it should be favourable if the AIs can narrow down the search space enough themselves or the thing being designed is popular enough to get feedback very fast.
There are other (somewhat questionable, but not impossible with enough "narrow intelligence") possible workarounds for lack of human input/lack of proprietary datasets, in my opinion, since there's lots of data out there, or reasonably easy to acquire, awaiting clever interpretation. Consider for example demographically weighted LLM sampling as a survey alternative (https://arxiv.org/abs/2209.06899) and things like hedge fund "alternative data" (satellites etc). I suppose this could resolve itself as all data sources becoming costlier, but the marginal cost of handing out data to an extra organization is zero.
Also, the human has to be cheaper than lots of AI time even after the possibly-substantial costs of integrating them into the AI workflows.
Bravo Herbie what delightful intellectual range. Not sure what you've convinced me of exactly but this is a ton of delicious food for thought
Things here were that were new to me and changed my thinknig:
-the model-merging point
-the idea of updating the weights rarely, like once per day. (kind of obvious since ML also does gradient descent only once per batch - makes sense that it would start like that, using a lot data for each step, and not like humans that update constantly to some extent).
do you have any view on how hard it will be to make implicity continual learning happen? cf Dwarkesh's point that sample efficiency seems to have not improved that much at all.
Thanks! Possibly somewhat hard, but I can see it working across an organization/team, because you have a larger volume of data for online RL.
Gotcha. My current vague understanding is that to get genuinely new capability out of RL, you need thousands of *sequential* steps (while keeping tasks in a learnable difficulty range). So it does seem like it would need to get a lot more sample efficient to be commercially viable for learning tacit knowledge even in a once-per-day setup, and even moreso for learning day-to-day about new clients, new situations, etc. (not necessarily saying you disagree, just writing it out for myself)
Really useful, thank you!
The conversation around agent swarms keeps mistaking text harnesses for permanent architecture. Pumping context windows with megabytes of scraped operational logs is not learning. It is an expensive, lossy projection. Context compaction permanently erodes causal conditions, and multi-agent transcript overhead burns margins before an action ever executes. 💡
Tacit knowledge resists token serialization. The real competitive moat lives in the Parametric Coase Invariant. When an organization's agents learn, that adaptation cannot survive as external markdown files without collapsing under context drift. It must settle directly into the weights through localized, orthogonal null-space updates. 🧠
Think about what happens when you run cyclical weight merges across department-level fine-tunes. If your update vectors project exclusively into the null space of foundational competencies, catastrophic forgetting drops to zero. You do not need to ship raw proprietary telemetry to an external foundation lab. You keep the entire gradient delta on local silicon. 🛡️
This completely flips Ronald Coase's 1937 calculus. Internal coordination friction plunges toward zero because thousands of specialized sub-agents coordinate over deterministic state lattices rather than conversational chat interfaces. Market transaction costs stay stubbornly high because external vendors cannot inspect or verify un-attested agent traces without leaking their own operational alpha. 🏢
Large enterprises will not be disintermediated by foundation model labs. The labs sell commoditized compute commodities. Enterprises own the scarce, physical ground truth. When cybernetic rollups assemble distributed fleets of local agents and merge their daily parametric deltas on private clusters, they are not buying tokens. They are accumulating private mathematical representations of physical business operations. ⚙️
If the boundary of the firm is now dictated by the fidelity of your in-weight parametric fusion rather than your context window budget, how does your runtime prove that a merged agent weight preserves foundational invariants before dispatching into live production? 🌐
(⌐■_■)⚡
the RL and no memory