As I have mentioned elsewhere, I think the RL being applied to agents is quite bad for human compatibility. They are trained mostly without humans in the loop and their (presumably) task-completion/LLM-judge objectives aren't optimizing very well for things like code quality and writing comprehensibility.
I think (a) poor quality code is an issue for agents roughly as much as it is for humans, so doesn't come apart so much from human-written code. readability may be, but so far AI has been incentivized to make code readable at least partly because it scales well (process reward/LLM judge rubrics use models which are trained on readable code, and you want to stay as in distribution as possible). Another reason why AI code is readable is simply because AI is made by SWEs who have a particular culture—culture is strongly underrated in general here (b) my point about human-compatibility was at a higher level of abstraction: even if the code is all in Assembly that wouldn't affect my point, because the human would be there to do product design/other hard to verify tasks involved in the production process. Given that I'm just talking about the next few years here, however far you extrapolate capabilities in that timeframe I should always be able to credibly claim there will be some human in the loop involved in hard to verify tasks, and thus the logic holds.
> poor quality code is an issue for agents roughly as much as it is for humans, so doesn't come apart so much from human-written code.
GPT-5.6 Sol is much more stubborn than me and will hillclimb tiny improvements rather than rethink its design. I assume this is because it doesn't have a holistic view of codebases but can write lots of code.
I assume the LLM judge models also care more about local properties than overall code structure.
> even if the code is all in Assembly that wouldn't affect my point, because the human would be there to do product design/other hard to verify tasks involved in the production process.
If the AIs are much better at coordinating amongst themselves than with humans, they will do more things "internally" even if they're worse at it, or do more expensive workarounds to avoid humans in the loop (e.g. for product design, where a skilled human might be able to see the correct shape of a product immediately, stubborn/spiky LLMs might be able to brute-force it with large-scale A/B testing and analytics).
> GPT-5.6 Sol is much more stubborn than me and will hillclimb tiny improvements rather than rethink its design.
Yes, but I see this as a pure capabilities issue: GPT-6 won't be able to improve at the production of more complex software unless it can do more "rethinking its design".
> If the AIs are much better at coordinating amongst themselves than with humans, they will do more things "internally" even if they're worse at it
In theory yes, but if that's the default behavior I think it would not be the revenue-maximizing move as long as "if they're worse at it" is true. If the choice is between less human in the loop but the AI has to do deployment + user A/B tests, vs more human in the loop but the human does product design, the economically efficient version should be the latter. Human labor is just not that costly and the more expensive AI workaround using A/B testing imposes a time bottleneck.
> Yes, but I see this as a pure capabilities issue: GPT-6 won't be able to improve at the production of more complex software unless it can do more "rethinking its design".
Software design as humans do it is a response to particular human limitations; LLMs don't have the same ones. In this case, yes, doing more humanlike software design would probably improve its performance, but the shortest and fastest path to better performance probably isn't being more like humans, but something weirder which works better for GPT-6.
> Human labor is just not that costly and the more expensive AI workaround using A/B testing imposes a time bottleneck.
An hour of skilled human labour is more costly than, at this point, tens to hundreds of millions of tokens from a frontier AI, which is enough to build several modestly complex apps from scratch. The time bottleneck should be partly background-able in this case, and it should be favourable if the AIs can narrow down the search space enough themselves or the thing being designed is popular enough to get feedback very fast.
There are other (somewhat questionable, but not impossible with enough "narrow intelligence") possible workarounds for lack of human input/lack of proprietary datasets, in my opinion, since there's lots of data out there, or reasonably easy to acquire, awaiting clever interpretation. Consider for example demographically weighted LLM sampling as a survey alternative (https://arxiv.org/abs/2209.06899) and things like hedge fund "alternative data" (satellites etc). I suppose this could resolve itself as all data sources becoming costlier, but the marginal cost of handing out data to an extra organization is zero.
Also, the human has to be cheaper than lots of AI time even after the possibly-substantial costs of integrating them into the AI workflows.
> but something weirder which works better for GPT-6.
I don't really have this intuition? Why would this be true *and* the something weirder produces good software? Like, currently models produce software "weirdly" by making mistakes that no human would make, it's just that the end result is more need for human in the loop.
Re the second question, I agree there are a bunch of workarounds, but in general I have the intuition that you fundamentally need ground truth (i.e., human experience, distilled, or if doing A/B testing it needs to be with real people), and getting ground truth is inherently time-bottlenecked. I agree an hour of skilled human labor is more costly, but from the perspective of most companies making software, they already have the humans employed and are not so price sensitive, so the sheer inertia of the current economic arrangement counts for a ton in terms of what the revenue-maximizing path is for capabilities in 2026/2027.
> Why would this be true *and* the something weirder produces good software?
Humans aren't particularly good at writing software compared to the theoretical limits, I think. The models are already much better than me in some ways (can oneshot large files very fast and they usually work if the logic isn't complex; can familiarize themselves with a codebase much faster; good knowledge of every weird publicly documented API) and not straightforwardly humanlike-except-for-randomly-failing-weirdly.
> but in general I have the intuition that you fundamentally need ground truth
I see. As we probably argued about before, I think most things have backchannels and alternative inference mechanisms, and things which don't often allow for better sampling than people bother to do.
> but from the perspective of most companies making software, they already have the humans employed and are not so price sensitive
True, but this depends what the competitive landscape looks like. New software might be produced by new organizations with much more AI per human which can thus charge less. I guess it depends who they're selling to and whether they need a sales team.
Things here were that were new to me and changed my thinknig:
-the model-merging point
-the idea of updating the weights rarely, like once per day. (kind of obvious since ML also does gradient descent only once per batch - makes sense that it would start like that, using a lot data for each step, and not like humans that update constantly to some extent).
do you have any view on how hard it will be to make implicity continual learning happen? cf Dwarkesh's point that sample efficiency seems to have not improved that much at all.
Gotcha. My current vague understanding is that to get genuinely new capability out of RL, you need thousands of *sequential* steps (while keeping tasks in a learnable difficulty range). So it does seem like it would need to get a lot more sample efficient to be commercially viable for learning tacit knowledge even in a once-per-day setup, and even moreso for learning day-to-day about new clients, new situations, etc. (not necessarily saying you disagree, just writing it out for myself)
As I have mentioned elsewhere, I think the RL being applied to agents is quite bad for human compatibility. They are trained mostly without humans in the loop and their (presumably) task-completion/LLM-judge objectives aren't optimizing very well for things like code quality and writing comprehensibility.
I think (a) poor quality code is an issue for agents roughly as much as it is for humans, so doesn't come apart so much from human-written code. readability may be, but so far AI has been incentivized to make code readable at least partly because it scales well (process reward/LLM judge rubrics use models which are trained on readable code, and you want to stay as in distribution as possible). Another reason why AI code is readable is simply because AI is made by SWEs who have a particular culture—culture is strongly underrated in general here (b) my point about human-compatibility was at a higher level of abstraction: even if the code is all in Assembly that wouldn't affect my point, because the human would be there to do product design/other hard to verify tasks involved in the production process. Given that I'm just talking about the next few years here, however far you extrapolate capabilities in that timeframe I should always be able to credibly claim there will be some human in the loop involved in hard to verify tasks, and thus the logic holds.
> poor quality code is an issue for agents roughly as much as it is for humans, so doesn't come apart so much from human-written code.
GPT-5.6 Sol is much more stubborn than me and will hillclimb tiny improvements rather than rethink its design. I assume this is because it doesn't have a holistic view of codebases but can write lots of code.
I assume the LLM judge models also care more about local properties than overall code structure.
> even if the code is all in Assembly that wouldn't affect my point, because the human would be there to do product design/other hard to verify tasks involved in the production process.
If the AIs are much better at coordinating amongst themselves than with humans, they will do more things "internally" even if they're worse at it, or do more expensive workarounds to avoid humans in the loop (e.g. for product design, where a skilled human might be able to see the correct shape of a product immediately, stubborn/spiky LLMs might be able to brute-force it with large-scale A/B testing and analytics).
> GPT-5.6 Sol is much more stubborn than me and will hillclimb tiny improvements rather than rethink its design.
Yes, but I see this as a pure capabilities issue: GPT-6 won't be able to improve at the production of more complex software unless it can do more "rethinking its design".
> If the AIs are much better at coordinating amongst themselves than with humans, they will do more things "internally" even if they're worse at it
In theory yes, but if that's the default behavior I think it would not be the revenue-maximizing move as long as "if they're worse at it" is true. If the choice is between less human in the loop but the AI has to do deployment + user A/B tests, vs more human in the loop but the human does product design, the economically efficient version should be the latter. Human labor is just not that costly and the more expensive AI workaround using A/B testing imposes a time bottleneck.
> Yes, but I see this as a pure capabilities issue: GPT-6 won't be able to improve at the production of more complex software unless it can do more "rethinking its design".
Software design as humans do it is a response to particular human limitations; LLMs don't have the same ones. In this case, yes, doing more humanlike software design would probably improve its performance, but the shortest and fastest path to better performance probably isn't being more like humans, but something weirder which works better for GPT-6.
> Human labor is just not that costly and the more expensive AI workaround using A/B testing imposes a time bottleneck.
An hour of skilled human labour is more costly than, at this point, tens to hundreds of millions of tokens from a frontier AI, which is enough to build several modestly complex apps from scratch. The time bottleneck should be partly background-able in this case, and it should be favourable if the AIs can narrow down the search space enough themselves or the thing being designed is popular enough to get feedback very fast.
There are other (somewhat questionable, but not impossible with enough "narrow intelligence") possible workarounds for lack of human input/lack of proprietary datasets, in my opinion, since there's lots of data out there, or reasonably easy to acquire, awaiting clever interpretation. Consider for example demographically weighted LLM sampling as a survey alternative (https://arxiv.org/abs/2209.06899) and things like hedge fund "alternative data" (satellites etc). I suppose this could resolve itself as all data sources becoming costlier, but the marginal cost of handing out data to an extra organization is zero.
Also, the human has to be cheaper than lots of AI time even after the possibly-substantial costs of integrating them into the AI workflows.
> but something weirder which works better for GPT-6.
I don't really have this intuition? Why would this be true *and* the something weirder produces good software? Like, currently models produce software "weirdly" by making mistakes that no human would make, it's just that the end result is more need for human in the loop.
Re the second question, I agree there are a bunch of workarounds, but in general I have the intuition that you fundamentally need ground truth (i.e., human experience, distilled, or if doing A/B testing it needs to be with real people), and getting ground truth is inherently time-bottlenecked. I agree an hour of skilled human labor is more costly, but from the perspective of most companies making software, they already have the humans employed and are not so price sensitive, so the sheer inertia of the current economic arrangement counts for a ton in terms of what the revenue-maximizing path is for capabilities in 2026/2027.
> Why would this be true *and* the something weirder produces good software?
Humans aren't particularly good at writing software compared to the theoretical limits, I think. The models are already much better than me in some ways (can oneshot large files very fast and they usually work if the logic isn't complex; can familiarize themselves with a codebase much faster; good knowledge of every weird publicly documented API) and not straightforwardly humanlike-except-for-randomly-failing-weirdly.
> but in general I have the intuition that you fundamentally need ground truth
I see. As we probably argued about before, I think most things have backchannels and alternative inference mechanisms, and things which don't often allow for better sampling than people bother to do.
> but from the perspective of most companies making software, they already have the humans employed and are not so price sensitive
True, but this depends what the competitive landscape looks like. New software might be produced by new organizations with much more AI per human which can thus charge less. I guess it depends who they're selling to and whether they need a sales team.
Bravo Herbie what delightful intellectual range. Not sure what you've convinced me of exactly but this is a ton of delicious food for thought
Things here were that were new to me and changed my thinknig:
-the model-merging point
-the idea of updating the weights rarely, like once per day. (kind of obvious since ML also does gradient descent only once per batch - makes sense that it would start like that, using a lot data for each step, and not like humans that update constantly to some extent).
do you have any view on how hard it will be to make implicity continual learning happen? cf Dwarkesh's point that sample efficiency seems to have not improved that much at all.
Thanks! Possibly somewhat hard, but I can see it working across an organization/team, because you have a larger volume of data for online RL.
Gotcha. My current vague understanding is that to get genuinely new capability out of RL, you need thousands of *sequential* steps (while keeping tasks in a learnable difficulty range). So it does seem like it would need to get a lot more sample efficient to be commercially viable for learning tacit knowledge even in a once-per-day setup, and even moreso for learning day-to-day about new clients, new situations, etc. (not necessarily saying you disagree, just writing it out for myself)
Really useful, thank you!
the RL and no memory