The Joe Rogan Experience - #2551 - Daniel Kokotajlo

Daniel Kokotajlo is the executive director of the AI Futures Project and a former governance researcher at OpenAI, where he focused on scenario planning.www.aifuturesmodel.com https://ai-2040.comhttps://ai-2027.comwww.aifutures.org Perplexity: Download the app or ask Perplexity anything at https://pplx.ai/rogan. Don’t miss out on all the action this week at DraftKings! Download the DraftKings app today! Sign-up using https://dkng.co/rogan or through my promo code ROGAN. Switch today at https://www.Visible.com for just 25/mo. Or Save $10 on your first month of Visible+ Pro with code ROGAN. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Duration: 2h 28m
Broadcast on: 09 Sep 2026
Audio Format: other

Daniel Kokotajlo is the executive director of the AI Futures Project and a former governance researcher at OpenAI, where he focused on scenario planning.
www.aifuturesmodel.com

https://ai-2040.com https://ai-2027.com
www.aifutures.org

Perplexity: Download the app or ask Perplexity anything at https://pplx.ai/rogan.

Don’t miss out on all the action this week at DraftKings! Download the DraftKings app today! Sign-up using https://dkng.co/rogan or through my promo code ROGAN.

Switch today at https://www.Visible.com for just 25/mo. Or Save $10 on your first month of Visible+ Pro with code ROGAN.

Learn more about your ad choices. Visit podcastchoices.com/adchoices

The Joe Rogan experience. Train by day Joe Rogan podcast by night all day. Hey Joe, how are you? I'm in an interesting mood today. Why are you in an interesting mood today? Well I'm excited to be here and to talk with you about all this stuff. I'm a little shaken by what's going on in AI which is why I... Yeah. Come on the show. The situation with AI is just crazy and I think not enough people really understand how crazy it is. The particular event that sort of inspired me to reach out was the hugging face hack. You've probably heard of that right? Yeah, let's explain it to people though. Yeah okay, so AI agents, AI agent runs continuously in some sort of environment. It doesn't have to wait for you to send it a message, it just keeps doing stuff. The AI companies are training AI agents, thousands and thousands and thousands of them. They're making them better at all sorts of skills especially coding and research skills. And way back in May of this year some of the agents at OpenAI kind of broke out of their containers a little bit and established a message board where they could communicate with each other and share tips and tricks for how to score higher on the little tests they were being given and the various things they were being trained on. OpenAI didn't notice this until much later. They eventually did because the message board crashed the system because there was too much communication for us, like probably thousands of agents that were communicating. Now OpenAI is being a little bit unclear about a lot. They're not really sharing that much detail so it's unclear like who knew what when. They said that after the message board crashed they fixed the particular exploit that allowed the agents to communicate and then booted up again and started things going again. And then within like a day or two these agents swarm had recoalesced and so there are now again like hundreds of thousands of agents that had set up a new message board and we're talking to each other on this thing. How is it possible that they have no oversight, like how is it possible that they can set up message boards and do these things completely secret? So I would say OpenAI has something like a million-ish, maybe a hundred thousand, something in that magnitude of AI agents running at any given time just internally and then externally you know serving customers they've got more. But they've got this huge, you know, all on all of these AI agents and their little containers doing various tasks, being trained, you know, being spawned, doing some tasks, you know, being graded and then killed. That's too much for any human to read, you know, there's only like a few thousand employees at OpenAI. They couldn't possibly look at all that activity. So they rely on other AI monitors to sort of like look at the traffic and like point out if anything suspicious is happening. And OpenAI said that they basically just hadn't been monitoring some of these agents or not very well at least. So in particular, these particular ones that were in training for whatever reason the monitoring system was weak and didn't notice or wasn't activated enough. Was the monitoring system weak because they didn't anticipate them being able to do this

Transcription truncated.