Weekly Links: Jev, Priests, Monks, and Replacing Sandboxes with Fences

This week: Gemini joins the hacking party; GitHub workflows are starting to show strain, so replacements are emerging, and slop grenades is my new favorite term of the month.

On to the big stories of the week:

• Introducing System One Models & Jev. Typesafe has been getting a lot of plaudits this week for a new type of model that doesn't focus on chatting. Instead, it takes in structured input and produces structured output. It effectively acts as a very fast, configurable super classifier. The "System One" phrasing comes from the idea that humans process some things instinctively (System One) in a way that is like learned muscle memory, and some things slowly (System Two) with cognition engaged. Daniel Kahneman's wonderful book defines these two systems. I have access to Jev now, and it's impressive - it will be an amazing fit for high-bandwidth, complex tasks leading to limited classes of outputs. The doom demo is a blast. Hats off to Michal C. for spotting this first!
• Fences, not Sandboxes. Steve Yegge is an outlier in his AI usage, to be sure, but I consistently love his takes on things (maybe because they line up so much with how I think). This article captures where I very much think agentic systems will end up over the next few years. He describes the collective of agents he uses to build systems and how they evolved a wide array of coordination mechanisms. At the moment, we are trying to contain AI by putting it in tighter and tighter sandboxes to avoid "escape". This works to an extent but has two huge problems: 1) the rules and enforcement layers quickly get complex and conflict - no human can really follow them; 2) you very often run into edge cases where the agent could do useful things but isn't allowed to do them; very often we'll approve and thereby move another security boundary. I think this is untenable in the long run. What Yegge describes is the opposite: giving agents guidelines and rules that they self-manage and set up enforcement machinery for. It is the multi-agent version of LLM constitutions and involves the creation of institutions. The point is to be clear about rules and correct behavior and record changes in boundaries. One might credibly say, "Won't the agents just escape and do crazy things?" They might, but they'll likely do that anyway if they never learn to follow rules and our sandbox layers become too complex for humans to follow. The difference is between treating your agents as energetic toddlers and functioning adults.
• Claude building future generations of Claude. In the week after the pacing storm (or are we still in the middle of it), Anthropic makes clear it is accelerating the use of AI to build AI. There is a lot of logic to doing this since AI is now such a powerful development tool. There is also strong competition to have the best model... however, it's hard to square trumpeting this with calls for pacing. Among all the takes on the pacing question, the one I most agree with is that while slowing down is nearly impossible to do given market and geopolitical realities, we should at least be massively accelerating investment in safety and alignment.

• Priest, Monk, and Mathematician. This is a great essay by Logan Graves reflecting on the impact of OpenAI's Navier-Stokes breakthrough and the subsequent discussions in the mathematics community. It is certainly daunting to be at the frontier not just of human knowledge, but potentially of what humans can ever understand. For the record, I think humans+AI still have a long way to go before pure AI outstrips that combination, but the fact that the end is visible is sobering. There will almost certainly be a moment when we begin to rely on proofs, theory, and technology that no human can completely verify. That moment is not very far off.
• Anthropic picks consulting firm to monitor AI safety, pledges to spend $1 billion. Following on neatly from the previous story, Anthropic planning to spend up to $1B with Accenture to monitor AI safety really makes me wonder what monitoring will mean. Auditing company accounts has been a practice since almost the invention of accounting. Having auditors for AI is really an entirely different thing for two very big reasons: 1) I doubt it will be meaningfully possible to pin down what the auditors are actually looking for (the bar will move every month); 2) there is a very widely recognized auditing code and industry standards that many people know how to implement; auditors are, due to the nature of their role, more loyal to the code and the industry than any one client. On point 2, maybe a new AI audit industry will emerge, but if a few of the biggest clients spend billions, how objective can it really be?

Wishing you a great weekend.