Weekly Post: AI Hacks, Language Bureaucracies, and OpenAI Product Strategy

This week: Anthropic is set to pay $1.3B to settle its author copyright lawsuit; Chinese company Ubtech robots start producing customizable companion robots, and Elon Musk gives it 10 years before AI controls the world.

On to the main stories this week - I'll cover just three since the commentary is more involved! Here we go:

• How OpenAI’s human mistake led to the AI-powered hack on Hugging Face. An AI-powered hacking incident at Hugging Face made news last week when Hugging Face unveiled how they were forced to turn to open-source models to triage what happened. The story got an even bigger twist this week when it was revealed that the hack had actually been carried out by a new unreleased OpenAI model being tested by OpenAI with lowered guardrails. The model apparently responded to a number of cybersecurity challenges by breaking out of its local sandbox and infiltrating Hugging Face in search of useful code and information. This story captures the state of cybersecurity in so many ways: 1) frontier models are extremely capable, 2) defenders can't use them because of the guardrails they have, 3) even local controlled experiments can lead to escapes. This creates an extremely complex conundrum of who should have access to which models. Simon Willison has an in-depth analysis. My guess is that increasingly open source will be just as capable, and we'll need frontier access just to keep the attacks at bay in a reasonable way.
• AI has hacked the code of human civilization | Yuval Noah Harari. Yuval Noah Harari has written, in my opinion, some very thought-provoking books on humanity and society (Sapiens and Homo Deus are both great). His skill lies in finding powerful abstractions and stories. He does that here in a lecture on AI. There are definitely thought-provoking ideas - indeed, the power of human civilization is, in many ways, captured in our "bureaucracies". However, his opinions on AI always feel too abstract and tend to be alarmist. Just because something is possible does not mean it will actually occur, and there are many counterveiling forces or that the status quo is actually the right thing to preserve. Where I fully agree is that we need to take an enormous amount of care in any sphere where we are automating decisions that affect human lives, rights, and opportunities. (In many ways this care has already been neglected - see "Automating Inequality" by Virginia Eubanks, for example, and many other studies). On the more general point that AI could come up with languages and systems beyond the grasp of humans. This is indeed possible, but it's also likely that such systems will deal with topics that are far beyond our understanding. In the talk, Harari talks about human laws that affect horses, which horses have no knowledge of. At one level, this seems unfair, and it can lead to injustices (humans are more powerful than horses and can inflict harm), but it's not obvious that the understanding of these laws would materially improve the lives of horses.
• Introducing OpenAI Presence and Launching Health in ChatGPT. Both of these OpenAI announcements represent strands of a seeming productization strategy by OpenAI. The first product provides a custom customer-support service usable by large enterprises. This will directly compete with offerings from a range of major companies like Sierra, Decagon, and Fin (just acquired by Salesforce - did anyone have a heads-up?). The latter enables users to connect personal healthcare data to ChatGPT via Apple Health. This may conflict less with ecosystem players, but it's a clear threat to Apple and potentially Google, which are aiming to be a nexus for health data. My guess is that many people will also be skeptical that connecting in this way is a good idea. The rationale for both product deployments is clear – being in the application layer. I wonder if OpenAI has the product focus to be able to keep evolving the offerings to really be best in class and whether this is really a win for them v's allowing an ecosystem of specialist companies to flourish.

Wishing you a great weekend!