Jacob Coxon's AI Doom, OpenAI's Maths Drama, and a Minecraft Fruit Fly

Justin: Hello, good morning, good afternoon, good evening, and welcome to the AI Argument. I’m sticking it out as our resident AI optimist, and we have, who’s on the ascendance, our AI pessimist, Frank. Frank, how’s your P doom, P gloom this week?

Frank: Oh yeah, PDoom is definitely spiking. Absolutely 100%.

Justin: All right.

Frank: But simultaneously, I do have a little bit of good news for you before we get kicked off. The AI Argument’s now officially a finalist in the Cork Chamber Digital Marketing Awards for 2026.

Justin: Woo-hoo. Fanta— And I can tell you that we’ve won already.

Frank: Is that right?

Justin: Claude’s on it. It’s doing it in the background. It’s gonna sort it out for us, gonna do things.

Frank: Okay, great. Is that now because Claude has weighed up the probabilities or because Claude has hacked into the Cork Chamber system?

Justin: I can neither confirm nor deny. Neither confirm nor deny.

Singularity update: are robots building robots?

Justin: So this week has been a mental week, right? And so, in order for this week’s show, I kind of have to take a deep breath, and I have to slow down. I’ve been totally overwhelmed by all of the things that have happened, and I’m going to assume that people that are with us on this journey haven’t been reading as much of the news, right?

So we’ll try and do—let’s try and slow us down, right? Before we…

Frank: As it happens, I took a few days off. So…

Justin: Okay.

Frank: I came back and was catching up and I was like, “What?” So yes, definitely. Walk us through what’s going on.

Justin: All right, so before we get into it, we’re gonna focus on two big stories, but as you know, for the last six weeks or so, we’ve been saying we’ve passed the singularity, the singularity’s here. So let’s just do our quick regular doo, doo, doo, doo, doo singularity update. So the quick-fire round before we get into the two big juicy, juicy stories.

The singularity’s here. Google DeepMind this week released Alpha Genome Atlas. This is an AI-powered searchable database of all of your DNA and what happens when you switch every single one of them within your DNA. So obviously, great applications there for DNA modification into the future. Insilico Med’s developed the first AI drug that is in phase three trials, so it’s not some airy-fairy thing.

It’s actually going through the whole process, and what it does is it reverses ageing. So that’s pretty cool, designed by AI. XPeng, who don’t make golf clubs for the more wealthy of those among us, but they’re actually a Chinese company that make robots. So they have, today or this week, launched the world’s first automated production line for advanced general purpose humanoid robots.

So these are robots that are gonna come and clean your house and make the tea and do all that sort of stuff. It is a factory with no humans. It’s just robots making robots, and it’s been made this week. So over the next year or two, we’re just gonna have loads of robots.

Frank: And that’s China as well, you said, is it?

Justin: Yes, that’s China.

Frank: And that’s your prediction from way back when we started coming true there about how you— At the time you were saying, “Look, China’s gonna have robots that can do a massive amount of manufacturing, and we need to figure this out in Europe or we’ll be left behind.”

I distinctly remember you saying that.

Justin: Every technology is dual use, and obviously robots that can clean your house can also wield a knife and do other things that maybe we don’t want them to do. So, you know, figure it out for yourself what other purposes, Taiwan, those robots might be put to.

And then a company called ORNL, researchers, they have developed an AI-guided system that can make any material on the planet atom by atom.

So we’re talking there about, what was that thing called in Star Trek where you could just make, you know…

Frank: Oh yeah. Yeah, yeah. Earl Grey, hot. Yeah.

Justin: Yeah, yeah, yeah, so it makes more sense. That is just the Singularity update. That is not the two main stories this week, but that’s the stuff that’s going on, and it’s happening faster and faster and faster.

So the first thing, Frank, that’s going on, the two really big…

Can ChatGPT Images get brand colours right yet?

Frank: Before—just very quickly before we get to that, because I have been a little bit out of the loop, so I missed a lot of those announcements, most of those announcements, in fact. But I was excited, just a small little, small little model release. I was excited to…

Justin: Oh yes.

Frank: ChatGPT release their updated image model in ChatGPT, and I’ve been playing around with it just a little bit, and it seems very, very cool.

It’s faster, which is great. And it is also—what I’ve found is it’s incredibly good at following very detailed, basically very long, very structured, very detailed prompts. Previously, you know, the last model was really good at doing that, but it would get stuff wrong, and you’d have to prompt again.

But now it just, so far anyway, in my experiments, I can give it something really complicated. I usually work with it conversationally, get it to write a structured prompt, then give it its own prompt, and the results are mind-blowing. Absolutely mind-blowing. The one thing I’m slightly disappointed about is it still doesn’t 100% nail things like brand colours.

They said, you know, it’s…

Justin: You’re so pernickety, Frank.

Frank: Well, I was hoping it might be the one because of the fact that they said, “Look, if you give it source material, if you give it,” for example, if I put a picture of myself in, it now changes even less details than it did previously. Like, there’s always been a bit of a shift, and it’s getting better and better and better at retaining, which is not good news for deepfakes, but let’s just point out.

But I thought, well, if it can retain characters, et cetera, can it also be precise about colours? And it’s still not quite there, which is a little bit annoying.

Justin: Okay, well look, we’re in the singularity. They’ll figure it out probably next week. So, you know, just hold your breath and it’ll be fine. Okay, so let’s go on to the two big, the two big stories…

Frank: Oh yeah.

Justin: …this week.

Frank: Oh yeah.

Is Jacob Coxon right about AI doom?

Justin: The first big story this week was the resignation of a security researcher.

Frank: Called Harry Potter.

Justin: Yes.

Frank: I was watching his interview. So I think his real name’s Jacob Coxon. Is it Jacob?

Justin: Yes.

Frank: I was…

Justin: AKA Harry Potter.

Frank: I was watching his interview and Marcy looked over my shoulder and said, “Why, why you, is that Harry Potter?” And since then I’ve seen all over the internet now him referred to as the Harry Potter whistleblower.

Justin: Oh, really?

Frank: Yeah.

Justin: That’s so funny. Okay, so we have a difference of opinion on this, I believe. So maybe you could just put forth your side of what you believe happened this week and why it matters.

Frank: Well, yeah. So Jacob Coxon basically tweeted that he was leaving Anthropic and he said, “Look, I’ve worked at Anthropic, I’ve worked at OpenAI. Neither company is coming at this responsibly, and they are building something that could conceivably kill us all in the future.” That was kind of the crux of it.

Justin: But there was a lot more to the story though, because it went viral very quickly, and he got a lot of support from a lot of other researchers.

Frank: Yes, including some fairly high-up people in Anthropic themselves who said things like, “Yep, you know, he’s right.” I reckon one of them, I’ve forgotten his name, quite high up, he said he reckoned that there was, like, an over 10% chance of AI killing us in, was it the next decade?

I can’t remember.

Justin: And it doesn’t matter. Okay, so my first, before we get into the nuts and bolts, my trite response is, well, if there’s a 10% chance that it’s gonna wipe us all out, there’s a 90% chance that we’re all gonna reach the sunny uplands, and if I have 90% chance of anything, I’ll take my umbrella, but, you know, I’m feeling pretty good about it.

It’s a chance I’m happy to take.

Frank: Yeah. Where’s our umbrella though?

Is someone orchestrating the AI doom narrative?

Justin: Okay, so let’s go and talk about Abrahams, right? So the things that were weird about this particular announcement, the things that sort of spiked my spidey senses was, one, when he announced his resignation on Twitter, he had not ever tweeted on Twitter before. It was his first tweet on Twitter.

Frank: Now, I read something interesting even about that, which I will admit is suspicious a little bit maybe. But what I read was that he had tweeted previously, but he scrubbed his account before doing this.

Justin: That’s possible too.

Frank: And well, if you know that you’re going to come under a lot of scrutiny, there could be lots of tweets you don’t want looked at. So I can see there being reasons. I do also accept it’s a little bit funny, it’s a little bit suspicious.

Justin: Okay. Not only that, right? So it was the only tweet on his account, right? It wasn’t a very well-known account. Ilya Sutskever, who was one of the best-known researchers in AI, when he resigned and tweeted about it from OpenAI in the middle of a huge furore and the ousting of Sam Altman, he got 6.6 million views on that particular tweet.

This particular tweet where he resigned has, I didn’t check it this morning, but yesterday it was at, like, 150 million views. It has got way more reach. It has been promoted, it has been amplified. It’s almost like it’s not organically growing. It’s almost like there is a hidden hand behind this that is pushing this tweet in order to make it more—get the message out there.

Frank: Now, Ilya did not resign saying that AI was going to kill us all. Like, he resigned saying, “I’m going off to make safe superintelligence.” So I do think—I take your point, but I do think that there is a big, big difference between the two examples.

But I will also say that the interview that I watched with Coxon, he didn’t deny that there was a certain level of orchestration behind this. Like, he very clearly said that he worked—this was not—like, when you read the Twitter thread, it comes across kind of like off the cuff and conversational and like, “Oh, I’m just bashing this out.”

But he said in the interview, like, it took him quite a while to compose this Twitter thread, and he wasn’t the only one involved. He said he got friends involved. They helped him compose the thread so that he could communicate what he wanted to communicate in the best way possible, and I am assuming in the most accessible way possible.

And he also said that he enlisted help so that when he tweeted it, it would get a bit of a push, people would retweet it, et cetera.

So he’s not denying there was a certain level of orchestration here.

Justin: Okay. So also last week, we had a video released by Sabine, and I’m not even gonna try to say her second name. It’s Hossenfelder. She is a physicist. She’s a scientist from Germany, and she does these great physics videos, and she’s now turned her hand to making AI videos, and she’s got loads of reach.

She has a YouTube channel. She released a video last week saying that she had been offered money to put out AI doomer videos, and in this she was being given specific words that she was to say in order to put across the message of AI doomerism, and she refused, and she said this was unethical, and she didn’t want to do it.

And so I’m not sure that what’s going on at the moment is unlinked to what Sabine has also been offered, right? There are people who have an agenda of AI doomerism, and behind the scenes they are pulling strings in order to amplify that message, and they’re backing it with money.

Frank: So I think you’re—I think there could be a link here, but I don’t think it’s the money. But I think there could be a link here because I did see somebody analyse—and unfortunately now I’m not gonna have the full details—but I did see someone analyse all of the retweets that Coxon was getting.

And they found, and I just don’t know how, you know, I haven’t looked into how seriously to take this, but they found links back to a lot of AI safety organisations, and most of those organisations, I think, linked back to, like, one particular, let’s call it AI doomer.

So they felt like this was orchestrated by an AI doomer, basically, essentially. Which is, like, it’s possible, right? It’s possible. I also don’t think that necessarily negates the message. And the reason I say I don’t think it’s about the money is because Coxon walked away from an AI research position at Anthropic.

That’s gotta be a very well-paid position right now, like very well-paid. And they’re, you know, possibly looking at going public, in which case I assume he would stand to make an awful lot more money. I don’t think any influencer is going to be paid for getting an AI doom message out there to the level that he could’ve been paid if he just stayed put.

Justin: Yeah, okay, that’s fair. Okay, I accept that, right? But, okay, I accept that.

Should powerful AI be open to everyone?

Justin: Let’s get down then to the meat of why I think this is just a problem, right? Why I think what he’s saying is wrong, why I think the outcome of what he’s saying I think is wrong, right, and dangerous. What he’s saying is that he believed there was a 10% chance that it could wipe us all out, and what he’s calling for is not just a pause, but he—I think he wants to stop AI development until we’ve figured out how we’re going to align AI so that, you know, we have good outcomes and we don’t have bad outcomes.

And I think the implication of what he’s saying therefore, you’re not gonna stop the NSA, you’re not gonna stop the CIA, you’re not gonna stop the big governments of the world, especially in the US and especially in China, in developing these models. So if you turned around tomorrow and you said, “We are not developing, we’re gonna have an international treaty that says we’re not gonna develop these models anymore,” you can be sure that there’s gonna be a project in the background where the governments are gonna fund the companies to create these models.

And so what you end up with is you end up with a three-tier system. You’ll end up with a system where normal people like you and me will get the dumbed-down models, right? And they’ll do all sorts of things to keep us entertained. There’s a thing like keep the masses entertained. I don’t know what that thing is.

But anyway, that’s what you will get, right? It’ll make funny pictures, it’ll do somewhat clever stuff and whatever, but it’ll be the dumbed-down version. That’s what you’re gonna get access to, and it’ll be really cheap. Then there’s gonna be a slightly smarter version, which is gonna be controlled by, I would say, they’re gonna put a gate in, a governance gate in.

So unless you are responsible by whatever terms we deem to be responsible, and so we’re gonna decide who the winners and losers are, you don’t get access to this special model, which is a bit smarter. That’s gonna be the second tier. And then there’s gonna be a third tier, which is the government tier, and nobody’s gonna know about it, right?

And you’re gonna have this tier. And so they can hack into your systems, especially you as an individual, because you’ve only got access to the dumb model, right? An individual stroke European, right, from a usual argument, ’cause we only will have access to the dumb models.

And, you know, things are gonna— You end up in this dystopian future where you got— You were on your way back from your European holiday during the week, right?

And there was a problem in the airspace, the computer systems in the UK shut down during the week. And it’s gonna happen at a time where there’s some other political thing going on, and then there’s gonna be rumours going, “Oh yeah, that was probably some dark web AI system that went in and ha—” And we won’t know because it’s all under the covers.

And so what I’m arguing for is transparency. I’m saying we should all have access to the same— The only way you can protect yourself is by having access to the same models. And the outcome of what these people are asking for, see, means that that will not happen.

Frank: So I hear what you’re saying, but, yeah, unsurprisingly, I don’t agree. I don’t think that the answer is everyone having access to the same models because I think the risks are just too great. I think there’s too many people out there who are gonna wanna do bad things with the models.

We saw a report from Anthropic recently, just very recently, about government, you know, foreign government people trying to use Claude for things that could lead to bioweapons and for surveilling their own people, et cetera.

And they’ve apparently blocked them, found them and blocked them, et cetera. And, like, that’s government people. Like there’s, you know, there’s lunatics out there who will use these tools to do bad things. And so I don’t think just giving everyone access— Like, okay, here’s something I was thinking about, right?

Let’s say I said to you, “Justin, let’s go on a road trip,” ’cause you drive.

Justin: Woo-hoo. Yeah.

Frank: So I don’t drive. I don’t drive. You drive. Great. Brilliant. So let’s go on a road trip. Brilliant. You’re up for it. Great. Now, if I said to you, “You know what I wanna do? I wanna travel across the States,” right? Brilliant.

Sounds like a good road trip. But here’s the thing, I want to do it in an articulated lorry. So could you rent us an articulated lorry to do the road trip?

Justin: Yeah, I’m up for that.

Frank: Yeah, but you couldn’t.

Justin: Okay, io ti.

Frank: Wouldn’t cover it, would it?

Justin: No, actually I don’t have an artic licence. Do they check for that sort of thing in America?

Frank: So, like, I don’t—in your first two tiers, I don’t see a big problem. I think it’s okay that people might need to be registered and licensed to use a larger model, and they would be very closely monitored in terms of what they were using it for. Then at your government level, I totally see the dangers, absolutely, 100%.

However, I think the EU has taken kind of the right approach, and we need to go further. We need to move faster. You know, we’re not perfect, but, you know, the EU AI Act prohibits certain uses even by governments, even by, you know, the military or the police. And I think that’s what we need to get to, and it’s not gonna be easy.

I’m not saying it’s easy, but I think we need to get to a point where maybe, yeah, we do have tiered levels, but the top levels with the governments are very, very closely monitored, and the AIs that they use are as carefully monitored internationally as the middle tier.

Justin: Yeah. I mean, I just don’t know.

Is a 10% AI extinction risk worth taking?

Justin: And, you see, the thing about it is, right, when we have this conversation, I think this goes to the very heart of the AI argument, right? And what people are worried about, right? So when you say you have this sort of—one of the criticisms I’m seeing online, and it’s true, is 10% is a made-up number.

Like, what’s— You’re saying 10%, but what’s it based on, right? You don’t really know. But let’s just say it’s 10%, right? It’s kind of a binary. It’s like yes or no, right? But anyway, let’s say it’s 10%.

And if it’s 10%, right, that it might kill us, I’m also looking at the 90% and I’m going, it’s 90% that it will cure cancer, 90% that it will cure ageing, 90% that it will cure, you know, climate change, 90% that it will cure starving and hunger and whatever, right?

The gains on the 90% are so huge, right? My brain looks at it and goes, “That’s a chance worth taking.” Like, that’s, you’re, that’s— You’re in a desert and you want to cross the river to go to the, you know, the Fertile Crescent on the other side. Let’s do it. There’s a chance the river might take us away, but let’s go for it.

Frank: I think that’s a really good point. But I don’t remember the name of the guy, but I’m pretty—I think we talked about him on the show. There was a guy I watched, and he was just saying, “Look, stop chasing superintelligence. Like, stop chasing the AI that can do everything and go back to narrow AI.”

So we keep getting these promises of this, like, bountiful future, and then, you know, we keep joking about it on the show, like Mark Zuckerberg, “Oh, yeah, great bountiful future, but right now we’re actually gonna take your job and just give it to AI, and you have no future.”

But what if instead of chasing superintelligence that could do everything, we just went narrow and we said, “You know what? Fine. We say AI can cure cancer. Let’s do that. Let’s pick a particular type of cancer. Let’s make an AI that can cure that.” Renewable energy, “Let’s pick a field of renewable energy. Let’s create a narrow AI that can solve that.”

There are ways forward, I think, to get to the positives without creating a superintelligence that could kill us all.

Justin: Okay. And we’re gonna move on to one of those narrow intelligences, right? But just to say for everybody, like, why is this so overwhelming? ’Cause, you know, we’ve been sort of discussing it fairly rationally here. This blew up online enormously because of all of the support that Jacob got across all of the other labs, where all of the people that should know are saying, “He’s right.

We do need to do something about this. This is serious.” And the timeframe was three years, if I’m not mistaken. His timeframe was three years. Or if not his, then some of the other researchers were like…

Frank: In terms…

Justin: The next three… The 10% chance.

Frank: Okay. Right, right.

Justin: Right. We’re not talking about something far away. That’s why it’s a big deal.

It’s like the researchers in the field are like, “Within the next number of years, not 10, not 20, we’re talking three to five, this technology…” Now, I ha—

Frank: Then the other side of the conversation—the other side of the argument then—is like Elon Musk saying, “This is a psyop,” and he basically, I think, basically thinks that this is Anthropic trying to get regulatory capture, basically trying to regulate AI so that they’re the only ones who are in power.

Justin: And to brighten up the conversation a little bit, right, before we move on to maths, right? Rune, who works with OpenAI, his point, and I agree with him, was that it’s incredibly hard to kill everybody on the planet, and maybe we’re underestimating how hard that is. I mean, if you blew up all the nuclear power stations, you’re still not killing everybody.

So, you know, to kill everybody is actually quite hard.

Frank: Yeah. I mean, my fear is not really that it—I think AI killing everyone is a simple way to get the message across that terrible things could happen. But to be honest, I’m not really worried about it killing us all, ’cause if it kills us all, we’re dead. What does it matter?

We’re gone. We’re not gonna know anything about it. What I’m worried about is, like, you know, what if it’s not death? What if it’s like a bioweapon that makes us all horribly sick and we don’t live any—we, we don’t live any shorter lives, but we live them in misery and pain and illness?

Justin: Ah, we’re fine here now. We could move to Leitrim. Nobody will get us there. Nobody lives there. Anyway, so that was amazing, right? So that’s all been blowing up. Then we have the incredible… So we have a bit of background.

Did OpenAI crack a 90-year-old maths problem?

Frank: So the producers, the producers got in touch with me again, Justin, and they said, “You know, recently, recently we had a Frank is Right segment, and this week we have a Justin Was Right segment.” So I’m fascinated. What’s this Justin Was Right segment?

Justin: Okay, so for at least six weeks I’ve been saying that we’ve crossed the singularity and that, you know, maths problem after maths problem after maths problem has been solved, and it’s only a matter of time before consequential maths problems start to get solved. So this week, right— Sorry, roll it back.

At the turn of the century, right, so in the year 2000, there was seven maths problems identified.

They were called the Millennium Prize maths problems. They were the seven most important maths problems to be solved by humanity in order to further, right? They were— Look, it’s maths, right? So it’s not stuff that you can hold and touch, but it’s maths. And they were the hardest maths problems, and there was a prize of a million dollars associated with each.

Now, in the last 24 years, right, these are maths problems that have existed for maybe a hundred years, right? We’ve known about them for a very long time. The prize has existed for 25 years. In the last 25 years, humans have solved one of them, just one out of the seven.

Frank: Right.

Justin: There was a rumour at the end of August, so what are we at now?

11th of August. There was a rumour three weeks ago that Anthropic were close to solving one of those problems, called the Navier-Stokes conjecture. It’s to do with, it’s to do with fluid, dynamic fluid flows, and it’s to do with the turbulence and stuff like that, right?

And so there was a rumour, just a rumour, that Anthropic were gonna go public, and the day before they went public, they were going to release a theorem that proved this particular millennium.

And OpenAI, this is what’s amazing about this story, OpenAI heard this rumour and said, “Ooh, maybe it’s possible that this problem can be solved.” They weren’t even working on it. On the 28th of August, they started to train a new model to have a go of these maths problems. It finished training on September the 1st, three days.

They then put that new model to work on solving the Navier-Stokes conjecture, and it ran for 84 hours. 10,000 agents apparently were put to work, and at the end of the 84 hours, it proved mathematically the Navier-Stokes conjecture, therefore, you know, entitling them to solving. So that now at this point is humans one, AI one, right?

So, but AI has been working on it essentially for six days. Humans have been working on it for 100 years. That’s why it’s a big deal. Now, it turns out then that there was a massive furore because OpenAI then contacted the people in Anthropic and said, “Oh look, we have this solution, you know, maybe you want to go first.”

And there was a big argument about who would get credit for solving the Navier-Stokes. And it turns out, the bells and the whistles, that they solved it in a slightly different way, and they didn’t exactly solve the same problem, and we’re not gonna get into it. But now today, there is a rumour that another millennium problem has been solved by OpenAI, and Sam Altman has released a tweet saying that— Not just a tweet, in fact, he also released a graph where they had a maths benchmark and they had their Astra on max thinking, and it was at, like, 0.2 on this maths benchmark, and the new model that they trained in three days was at 0.6.

So we’re gonna see many, many more of these millennium and more problems be solved very, very quickly.

Frank: So the—I mean, there’s so much here that’s mind-blowing. Like, training a model in a matter of days is crazy, and then solving a problem in, like, whatever it was, 88 hours, that’s mental as well.

Did OpenAI train on a mathematician’s prompts?

Frank: But I do wanna dive into the soap opera aspect of this a little bit because the—you mentioned that they heard this rumour that one of the problems was being solved.

And so there was two mathematicians, Buck—Tristan Buckmaster and Levent Alpoge. I’m not sure how to pronounce his name. But Levent…

Justin: Yep.

Frank: He works at Anthropic, so he’s the Anthropic guy you were talking about. And OpenAI heard a rumour they were working on one of the millennium problems. Then apparently OpenAI started putting their new mathematically capable model, basically put them on all of the millennium problems.

They then identified— They solved— They had a breakthrough. They solved something, I don’t know, the Euler proof or something…

Justin: Yes, it’s related. Yeah.

Frank: …which weirdly was the same breakthrough that Buckmaster and Levent had had, which then led OpenAI to go, “Okay, with that proof, the…” What was it called? The, the Navier…

Justin: Navier-Stokes.

Frank: The Navier-Stokes conjecture is gonna be the easiest problem to solve now that we’ve had this breakthrough.

And the problem was that Buckmaster was like, “Hang on, this is really weird. OpenAI are saying that they’ve solved this thing, but it sounds like they’ve solved it in a very similar way to the way we have.” And Buckmaster and Levent have been using Codex, and so Buckmaster reached out to OpenAI and said, “Is there— Basically, is there any possibility that your model either used input that we put into Codex or was trained on input that we put into Codex?”

And initially, OpenAI first came back and said, “No, we don’t look at user data.” And he was like, “Okay, but you haven’t answered my second question. Could it have been trained on my data?” And they said, “Well, we can’t completely rule it out.”

Now, this was interesting because it, like, caused a huge, you know, question mark over, like, had Buckmaster and Levent been using OpenAI…

Justin: This is hilarious. We’re gonna…

Frank: …off the training data, the training— You know, had they not opted out of training data?

And that’s not clear, so which is interesting.

Justin: This is hilarious because we’re now switching our conspiratorial hats.

Frank: Yes. Yeah, yeah, yeah.

Justin: Your conspiratorial hat on, right? So the exact response—’cause when I saw this first, I was like, “Oh,” right? And I’ll tell you what I thought was happening. So the exact response that they said was, “While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.”

And so I read that first, and I’m going, that sounds to me like they take your inputs, and even if you’ve said, “I don’t want you to train on my data,” they change it in a way that’s mathematically irreversible. That would be the definition, right? So I can never get back to your original prompt, but I do kind of get your intellectual capital, and I can then use that to train my models.

Now—

Frank: That statement was really dodgy ’cause they’ve opened up a whole can of worms here where people now are saying, “Well, does this toggle actually opt me out of training data or does it not?” Like, it—

Justin: So I was reading, so there’s a guy that I interact with on Twitter called Prince. He is a lawyer. He’s from California. He’s very up on the AI, right, field. I would totally trust what he says. He’s very, very reputable.

And he came back and said, “Well, as a lawyer speaking here, that’s exactly how I would tell you to respond to that question,” because you cannot say, right, when you’re asked the question first, you can’t say with 100% certainty that we didn’t do it, right?

You’ve got to go and check. So you give a sort of a fairly vague answer first to cover yourself, then you go do the checking, and then you come out afterwards and give the definitive answer.

Frank: Which is what has happened. Yes, they did come out yesterday, just yesterday they came out and they said, “Look, we’ve looked into it and we did not use any of your data in any which way whatsoever.” And as far as they’re concerned, case closed.

Even though, right— So it does, I think it does kind of mean that people are questioning exactly what happens to their data again now, which is not great for OpenAI, I mean.

But even if we put that aside for a second, just the fact, because when Buckmaster got in touch with OpenAI, they gave him that information and they said they had solved Navier-Stokes, and they said, “Look, you—we have two options. You could publish on the Euler thingamajig that led to it, and then we publish Navier-Stokes.

Or you can publish Navier-Stokes, give us some credit, but you have to take Levent Alperge off the paper because he works at Anthropic.” So, like, one way or the other, OpenAI have been a bit dodgy about this, and even the fact that they heard the rumour and then swooped in to try and solve the problems before the human mathematicians doesn’t really align with their whole, “Oh, we want to support mathematicians, and, you know, we want to empower and enable mathematicians.”

Justin: Okay. And if you, you know, if you wonder why, why do I think this was such a heavy week and such a wow week, let me refer you to our previous conversation where we had the safety researchers say, “We need to slow down. We need to…” Right?

Look at what’s happened here in the field of maths. You had researchers from different companies going hammer and tongs to beat each other to get to a goal, and they couldn’t even agree on the academic publishing of a paper.

There is no way that there’s going to be an agreement. Very, very hard to see that there’s going to be an agreement between these companies to slow down if they can’t even agree on publishing a maths conjecture proof.

Frank: Very true.

Can brute force become superintelligence?

Frank: There’s one other thing that I think is interesting about this as well. So just in terms of what you were saying about the singularity and about the, you know, solving this in 88 days—or 88, sorry, 88 hours. I do think OpenAI have been a bit shady about this. I, you know, but I also think that they may, they may not really…

Sorry. What’s interesting to me is the original mathematicians have said that they actually were working on work that was started by two other mathematicians, Diego Cordova and Luis Martinez Zorilla. So that work pre-existed.

Now, if you consider OpenAI have got this model that they’ve trained specifically for maths problems, and they then put a swarm of 10,000 agents using this model at this problem, and first they started with all of the millennium problems, then they narrowed it down to the most likely one, and then they put all the resources on that one problem.

So it’s not inconceivable that 10,000 agents go through all of the existing materials on this problem and come to the same conclusion that Buckmaster and Levant did, which is, yeah, the work by Cordova and Martinez Zorilla is the most plausible way forward, and then the 10,000 agents move forward on that and solve it.

And the two papers, according to OpenAI, the two papers are different.

Justin: They are different.

Frank: So, like, this just again goes back to the thing of, like, to me, it’s this thing of, like, the swarm of 10,000 agents. It’s like superintelligence might not be a single model that is insanely intelligent. Superintelligence might be tens or hundreds of thousands of agents trying every conceivable thing until it just…

So you know what I mean? It’s like, it’s almost like…

Justin: I do, I do. And I was…

Frank: …of luck.

Justin: I was about to move on to the fruit flies, but I do have to stop, right? So, couple of things on the— So if you remember, go back to AlphaGo, AlphaZero, right? So what history has taught us and all of the models that we had before, if you look at token efficiency and chains of thought and so on, the first time it’s brute force, and then you reinforcement learn on the brute force in order…

And then the second time and the third time, it’s not brute force because now that knowledge has been encapsulated inside the model. So you’re kind of right, but it doesn’t matter because it means the next iteration of the model isn’t going to brute force it now that knowledge has become part of the model, and you move on to the next problem.

So it’s, to me, it’s like, yeah, whatever. I think maths is, in a sense, it’s solved. It’s just, like, even if you think about it, you say, “Oh, well, maybe we could’ve got 10,000 mathematicians and we could have stuck them in a room if they were all…” It’s like, well, that doesn’t matter. The AI can have 100,000 mathematicians.

Like, you can now solve a 100-year-old problem in three days. That’s— So even if it’s not novel, even if it’s brute force…

Frank: For a cost…

Justin: …ability to s—

Frank: 15—for a cost of about 15 to 20 million, it should be said as well, just to contextualise it.

Justin: Was what I read, but yeah, whatever, I’ll take 15. Doesn’t matter. But they won a million dollars, so, you know, he got some of the money back, right? And they got a better model, so that’s fine.

But look at— It is— So the things, these things that they are solving, so Navier-Stokes, right, why does it matter?

Right? Well, what difference does it make, right? I’ll give you one example, and there’ll be loads, right? It is a purely maths problem that they’ve solved, but it’s to do with fluid dynamics, and it’s to do with fluid dynamics moving at speed, and it’s to do with turbulence.

If you want fusion power, so instead of dirty nuclear power stations, we’ll have clean nuclear power stations, right?

They need to swirl plasma really fast in a container, and they need to contain that plasma in order to generate heat to generate power. This could potentially help increase the speed at which we achieve that goal. That’s the sort of stuff why it matters, and that’s why, you know, I think that this is important and, you know, it makes a difference.

Why is a virtual fruit fly brain playing Minecraft?

Justin: Anyway, if we didn’t have a data centre full of 10,000 brilliant scientists solving really hard maths problems, we could have already a data centre full of fruit flies. Excited news, Frank.

Frank: Sorry, what?

Justin: So you know and I know, right, that a number of years ago, we had this project to create the brain of a worm, and we managed to recreate the brain of a worm, and you could have a worm in a virtual world. This week, now have—scientists have released the brain of a fruit fly.

Not a, like… And I’m not even joking.

This is literally the connectome of a fruit fly, so this is the brain and nervous system of a fruit fly in a virtual world. And they’ve released it, and a scientist put it into Minecraft, and they got the fruit fly to start flying around in Minecraft, and that was kind of cool.

So then people said, “Well, that’s cool ’cause you can download the fruit fly’s brain, and you can start to do things with it.”

So people have put these fruit flies to lots of fun and entertaining tasks, such as playing Beatbox, that game where they get the fruit fly, or they give it two, like, drumsticks and it’s gotta hit the right notes. They had it playing Doom badly. One person has given it money, and it’s now playing the stock market for him.

You can train the fruit fly. Basically, you could do refinement training on the fruit fly by just, you can get it to do things, and you fire its endorphin receptors to give it a good feeling when it does the right thing.

Frank: So could you train a real fruit fly to play Minecraft?

Justin: Sorry, you could at— but badly. It’s really, really bad. It turns out fruit flies are terrible at “Minecraft,” and they’re terrible at “Doom.” And look, if you want to know how the superintelligence is gonna treat it… I mean, okay, so this is all fine, well, and good when it’s a fruit fly, but, you know, obviously they’re gonna do a rat or a mouse next.

Is that okay? And then after that, they’re gonna do a dog. Is it okay to put a dog in a virtual world and do weird things to it? And then what about when they do the ape, right? And then when it goes from an ape to, like, a human ape, is that okay to do weird things in a virtual… I don’t know. It’s got a big question over it.

But if you wanna know how the superintelligence is gonna treat us, maybe we should be looking at how we’re treating the fruit fly in the computer.

Frank: Well, and, you know, it doesn’t sound too bad, right? Playing computer games.

Justin: You wanna see some of the other people, what they have done? It’s not good.

Frank: Oh, okay. Okay. Maybe… Okay.

Justin: If you want to know where the 10% doom is coming from, it’s coming from the fruit fly. They’re coming back to get us.

Frank: I’m not sure. I think I’ll just… I’m not gonna Google that. I’m just gonna stick with this idea in my head that the AI superintelligence takes over the world and just leaves us humans to play “Doom” and “Minecraft,” and I’ll be happy enough with that, Justin.

Justin: I’m with you. I’m with you. Yeah, absolutely. All right, Frank, great week. Let’s see what…

Frank: Chat to you next week.

Justin: Talk soon.