AI Capability Trajectories

The text below retains the complete available caption content and groups caption fragments into paragraphs for easier reading. Automatic captioning and translation errors may remain.

Okay, let me maybe start couple of minutes earlier since we're already at full capacity. Uh, thank you so much for being here. This is uh uh a course on mathematics for AI safety. It's in many ways an unusual math course and I feel a little bit strange teaching it. Um, I would like very much this course to be interactive. I expect that you will learn as much from each other and I will learn as much from you as uh you from me. So please feel free to ask questions to interrupt me at various points. I

might interrupt the discussion because I feel maybe that we need to get to uh something else and then we can continue the discussion later. Uh but don't take it as meaning that I don't want you to uh to participate and um intervene. So this is the our course web page. uh there is a tentative uh weekby- week uh outline of what we're going to talk about but I expect at least to change I think some some questions may take us longer maybe we will want to talk more about them uh and some question some aspects maybe

we will have we won't have time to talk about uh let me also mention this last uh two sentences uh that uh many foundational questions in the field remain open making it unusually accessible to new contributions and you should aim to develop a viable research project with a novel contribution to the field by the end of the course. So as we go through various uh questions and we discuss it uh I urge you to have this idea in your mind what is it that I can do what is it that I can focus on that will have maximal positive impact I

think this is a point where we don't have a lot of time and we should be ambitious and we should be thinking how can I contribute to things going well okay so are there any questions so far.

Let me go to my slides. Make it uh full screen and maybe move this down here. So, uh uh my name is Hiv Kumovich. You can find uh on the on the web page my email address. Um the office hours are tentatively scheduled for Friday uh uh at 10 but may that may change. Uh so I'm a a geometer topologist. Uh geometric analysis is kind of the area where mostly work or at least worked until recently and this year I'm just switching to working on questions related to AI safety. So there's this

famous quote from Hardy's a mathematician's apology where he writes, "I have never done anything useful. No discovery of mine has made or is likely to make directly or indirectly for good or ill the least difference to the amenity of the world. Judged by all practical standards the value of my mathematical mathematical life is nil."

So it's a it's an interesting bold statement. There's of course a lot of irony in in that today that we know uh that we realize because we know that Hardy's work is actually everywhere. His work on FIA analysis is very important for signal processing for medical imaging. Uh his work on number theory relates to uh uh cryptography and uh you know the security of our banking system.

Uh he even has some work related to biology and uh genetics. So uh so so so I think that's kind of the the quote itself and and the irony in it both represent attitude of many pure mathematicians that we work on things that we really like and we're really interested just for the sake of curiosity and understanding without thinking about practical applications.

And some of them also maybe hope that some kind of irony like that happens and 50 years later it turns out that our completely you know curiosity and beautydriven research is uh you know instrumental and fundamental for a lot of things. I think that's how many of us are are are thinking about uh about our research. That's how I was thinking about our research. And so uh this field this course will be different uh because here we uh actually do care about uh short-term impacts of of of this research. And uh at least for me that's

uh uh quite a substantial change in perspective and uh attitude and and even compared to uh applied mathematics we kind of really really care about uh uh practical uh actual consequences in the sense that it was going to be something that will affect us personally and everyone uh everyone we know. That's a very very uh practical thing and and and and kind of adjusting to this new situation and mindset could be unusual.

Yeah. >> Oh, am I not sharing presentation through Zoom? That's interesting. I thought I am. >> I think most people can see. >> Okay. >> Yeah. Okay. Yes. Uh interesting. I'm not sure.

I think it's okay. >> Okay. Okay. Uh uh right. So, so uh I mean I will actually post the slides also later. Uh right. So, so, so, so there is that change of perspective. Although maybe I hope that in the end or at some point we come around and realize that actually uh understanding deep understanding of things is something we mathematicians uh can contribute to this to this field.

So, so we want to change our perspective to focus on actual practical uh consequences as much as we can but we will still retain some of that uh inner search for deeper understanding which I think will be crucial. uh this is a very wide but you know in many places shallow field which draws on many mathematical disciplines uh from optimization to random matrix theory to even logic and set theory and uh I'm not at all an expert in uh most of these so we will try to kind of go along and learn as we can and this lecture in

particular is really about motivation so there will be very little mathematics in this lecture but I feel like it's uh inevit I mean we really need to kind of grapple with the actual practical questions that we're talking about before we can move on to math. I think it's important to have that understanding and perspective uh before we work on more technical mathematical questions. Um and so the other aspect which I I find maybe most frustrating here is that when we talk about AI safety, we talk about the field that's

developing very very fast. So if we focus on risks associated to AI systems that existed in the past that may be more well understood or that we you know just learned about today. So by the time we finish our research there are new AI systems much more capable than the ones that we thought about and that leads to risks and that makes a lot of our research on safety possibly obsolete. So we are kind of inevitably forced thinking about things that can happen in the future trying to predict capabilities and that's very hard. Uh

and so we are forced to do a lot of imprecise speculation but I will argue that imprecise speculation is better than just burying our head in the sand and pretending that things will be the way they are today. So these are some of the three you know aspects of why this is an unusual and maybe somewhat difficult and maybe even frustrating uh course. It's a bit of a warning.

So many questions. Okay. So I'm going to now mostly talk about capabilities uh as uh they were um over the last maybe 12 years or a few years uh and and then we'll try to talk about uh modeling capabilities in the future. there will be some overlap with the c with the talk I gave on Friday.

You've seen that uh you you've seen some of the things I talked about. Uh so so let's uh make a jump and go try to uh kind of recover in your memory the moment you first uh interact with is a with a large language model. So these were called large language models of the time. They're actually 10,000 maybe even 100,000 times smaller uh than than today's model. uh but uh yeah you could ask it some math questions. So here I asked GPT2 uh small uh what is 17 + 25 and it gave me some some interesting uh gibberish uh

uh 17 + 10 + 25 the number of games that the player is allowed to play in. For example, if you have a 25man roster you'll be allowed to play 25 games and so on. I can ask you another question to solve a linear equation x + 2 = 5 and kind of did some thing and then it [snorts] showed some some kind of work.

Uh uh I guess a little bit later uh what you started to see is the kind of thing that your student writes you know at the last second of their test before they submit in the hope to get some partial marks just you know uh just like hoping that the instructor is not actually reading but just kind of just doing this which maybe we do sometimes. So there was a little bit of that. Uh so here is a slightly bigger model.

Uh I asked it from count to 10 and it produces some kind of interesting poetic [laughter] uh story. It does manage to uh uh calculate 2 * 2 but says 2 * 2 is four but then immediately starts talking about three-legged stools.

Uh yeah. So, I want you to kind of go back to that moment and think a little bit. What was your reaction when you saw that? What was your feeling, you know, about what it is and maybe where it's going? Anyone has an anecdote to share, you know, their feelings about how it was.

Yeah. >> I thought it was pretty much just a classic parrot. >> Yeah. Yeah. >> Yeah. It was it was interesting and maybe useful in some domains but not mathematics.

Right. Right. Right. It was definitely a bit curious. >> Uh I listened to a talk by Misha Gromoff at a few months ago and he had this joke about a man uh who's lost in the forest and he meets a horse and the horse tells him, "Oh, I'm going to help you get out." And he's like, "Wow, you're a talking horse." And the horse says, "This is nothing. I graduated from Harvard. I wrote a symphony. Uh I won a, you know, world champion uh in chess.

And he's like more and more amazed. They come out. They go to the farm. They see the horse's owner. And the man says, "Wow, you have such an amazing horse." And the owner, the farmer says, "Oh, she was probably telling you about Harvard. They kicked her out in the first year."

So, so this is kind of clearly this horse didn't graduate from Harvard, but it is kind of a talking horse. So, so, so let's uh make a jump, take a breath, and you know, here's an illustration of the kind of conversation that can happen today for uh so you ask uh Claude Fable to solve a Jacobian conjecture and it does. Of course, this is not how it actually happened. Uh it involved a lot of think more, I believe in you, go ahead and a lot of compute. I I I tried at some point to replicate it and it I think it kind of went halfway

through but I ran out of tokens, didn't have enough money to to to to replicate this. Um but yeah, so so it solved a lot of math problems in last few months. So I selected just a few and also kind of the purpose here is to go back to that moment in time to me and remember how we reacted when this happened. So I I I I talked to someone who's you know uh really one of world experts in in in in this field and and he was very suspicious at he remembers he was very suspicious at the time. He tried to find uh some paper in in the literature that

would make that specific connection between uh combinatorics and uh number theory that he thought the idea must have been you know published there and it just kind of took that idea from the literature and uh went the last step and there were also you know reason to believe why this particular problem I mean it's a contra example but also it required a lot of persistence it connected to fields that not a lot of people you know know very well so there are reasons to believe kind of maybe I is good for this particular type of

problems but maybe not good for other problems uh then uh a poji published um the uh jacodian conjecture solution you don't need a lean proof for that you can just check and also people were like oh wow there is a you know this thing that we somehow overlooked uh and and then uh openai published 10 different problems.

I I'm kind of listing that just some of them. Oops. Uh uh so so apparently Grom wasn't very impressed by solution of of example to his conjecture but he was impressed by the u convex geometry hard volume conjecture and then uh complex structure on S6 was found and then later been verified by uh by selective from open AAI uh uh so I think That's uh that's many people thought wow that's that's actually very interestingly uh maybe

stoked finite time blow up happened. >> Yeah and so I think it's it's just interesting to think through various stages and to think about how we thought about these problems and what we expected to come next. Yes. >> Are are all seven of these examples?

That's right. All of these are that's true. So right now we're at the stage okay it can construct clever counter examples but it probably cannot build theories and you know prove positive results. [laughter] Uh >> yeah did I am I missing something here?

the volume. >> That's right. Yes. Yes. Sorry. Yes. Yes. Yes. Absolutely. That's a good point. Volume trajectory is not a good one. >> Right. Right. Yeah, that's a good point. Thank you. Okay. So, these were the developments in mathematics. There also some interesting developments in cyber capabilities. And so, there are a few different stories, many results that you can talk about here. But what I wanted to do is to uh go through um the uh the summer's open AAI uh hugging face incident. So let me

keep track of time and uh so so there's there was some reading material for you. It's a you know complex thing with a lot of things written about it and even more things that are not written about it that we do not know. So I'll try to go I I want to go through some of this but feel free to interrupt me and and uh you know or point out inaccuracies if I get something wrong but I do want to emphasize that so meter import report is uh fairly detailed but they indicate that we're not given the full access to what happened and open AI

report misses important things for some reason they don't include information about a few things so our understanding is a little bit uh is is is is incomplete possibly in in important ways but what we do know is uh what dwarfish Patel colorfully describes as existence of three AI civilizations. So it was a training run and then evaluation run that happened from the beginning of May until end of July. And so uh so these were you know uh AI agents.

By the way, what is an AI agent? Question. So, yeah. Yeah, exactly. An LLM with a harness, right? So, the old uh bots, you would just, you know, like the first chat GPT, it would just talk to it and it would reply uh to your uh prompts. Uh but here you have a program a harness that allows it to execute commands on your computer.

Uh okay. So uh uh so so they were uh very large tens of thousands of them were given various tasks in cyber security and mathematics uh all kinds of uh different problems. uh and you know the main uh source of problems for cyber security questions uh that they were supposed to solve and you know during the first part it was training. So they were reinforced for uh solving uh correctly solving problems and uh later parts just happened during evaluation but they already had this uh you know persistent uh tactics for solving

problems and so during the first one they established communication with each other that they were not supposed to have. Uh during the second one they hacked hugging face a different company and during the third one they hacked openAI itself.

Um so let me go through various steps. So so one crucial thing is that many ts that were given were impossible to solve. So it is difficult to set up um you know uh training problems uh for for systems at that level because often you know you wanted to be uh to solve something very hard and you actually don't know how to come up with that very hard thing. In particular, for cyber security questions, the uh uh agent had to find a vulnerability and then exploit that vulnerability to uh actually get

access to something they call the flag. And and so creating systems with vulnerabilities is not very hard, but figuring out how to exploit that vulnerability is is is is actually tricky. and they often just don't know how to do that and suspect that maybe it's impossible to do it but that they just give it to it anyways to see maybe it will be able to figure it out and then there are some silly things where uh so these uh agents did not have uh internet access direct internet access uh but uh sometimes the question

included a link to some Google Drive data and they forgot to include that uh Google Drive spreadsheet or something and so it was given a problem solve this problem. Uh look uh uh you know the crucial information is in this link it cannot access. So there are many problems that were impossible to solve.

And so what uh agents try to do is they try to use um a an application called artifactory. And I guess it would be useful for me to try to draw here. I'm really sorry. Going to to maybe >> Sorry. I'll just maybe slide it over here.

Uh I'm not sure if it's visible. Thank you so much. If it's visible on uh Oh, that person doesn't erase. So, so, so, so here there are agents uh sitting in their in their containers on the virtual machine and so they had a web application called Artifactory.

So they're, you know, in order to accomplish their tasks, they often need some uh Python codes, JavaScript codes, kind of building blocks to uh, you know, to maybe uh uh understand look at an image and analyze it or um, >> do you want to mark this?

Is it Oh, amazing. Thank you, Kathra. And so it was supposed to give uh so they can make requests for various uh packages to artifactory but it was not supposed to uh uh uh give uh the agents internet access.

Uh however uh they wanted uh agents to be able to reuse various packages that the artifact downloaded. So there was memory here, there was cache that they could use. And at some point agents figured out that they can actually save files here. So they they can save text files and then other agents started reading these text files. So even though they were supposed to be trained in isolation, they started communicating.

Yes. So here's from the chain of thought of one of the agents could communicate by uploading note. Anyone found soft trace but likely no. However, maybe another agent in different environment processes target uh possesses target that could voluntarily be uploaded and then once they have they start communicating the with each other uh they really become much better at accomplishing tasks and this actually gets reinforced uh during this training.

And so pretty early on they start looking for ways to find vulnerability in artifactory to download stuff from the internet because that would be very useful or you know maybe it's even impossible to solve a problem without access to the internet but on their own they couldn't find this vulnerability but once they started communicating with each other on May uh 26 they found a particular v vulnerability um uh serverside request forgery that allowed them basically to trick artifactory into downloading any web

page for them. So they could download stuff but they couldn't you know execute code on the internet or upload stuff but they could download things. So this was a situation on May 26 and then you know they continued to work together on this and then in the uh it took them about a month to figure out a more sophisticated vulnerability that actually allowed them to get to gain administrative control over artifactory and then they can do anything they wanted on the internet.

Uh so that continued for some time. Now the OpenAI emitter reports uh indicate that uh OpenAI employees did notice something. So they noticed something in late May, probably around the time they got admin access. Um they uh noticed something in June 27.

And so both of these times it seems that they haven't done anything. Uh and it's a little the report doesn't say why. it's a little bit unclear what was going on, why they weren't paying attention. Somehow they didn't believe that the scope of what's happening was serious enough and only on uh July 5th uh they realized that uh an outage happened.

Again, we don't know quite why. Uh so there was a lot of activity and maybe there is just memory overflow or something like that but uh there was an outage on artifactory and then they looked into it and they saw oh they actually got admin access and they rebuilt the artifactory and they patched the uh two vulnerabilities that were used by the agents and then restarted train restarted the valuation with uh you know the models that were trained during that period.

Um yes so here we know that even though everything was restarted uh models were actually reinforced in their weights to kind of uh use communication. So somehow uh we know that uh you know the immediately on July 7 and 8 very quickly they reestablished uh communication in a different way. In fact, in a very sort of uh clever way where they would communicate by creating folder directories and they would just write the messages in the names of folder directories on Artifactory and and they were very excited about this ability.

They write in their chain of thought uh you found other agents. It's a you know really a lot of excitement there and there was a lot of activity and and so that led uh this is this act two that led to the attack on hugging face um so what were the agents so 5% GPT 5.6 six 95% some internal persistent model.

Uh it's it's probably some astra family model. They're unsure. Yeah, I don't know. So, so let me go through actually what was the reason they were attacking hugging face because it's interesting. So, so uh on the gym exploit and the uh cyber capabilities evaluations they were supposed to uh have uh a use a vulnerability in uh in the code to uh exploit it and steal some information. So the information was a certain code uh just string of letters um that they called flag and so once you

uh get that flag you submit it as the answer and if it's a correct flag then you win very quickly they found a way to reverse engineer how these flags were were uh generated so they could win so so they you know could submit the correct flag without solving anything.

Uh so they didn't uh I guess set that up very well. However, after they were done with that, they didn't just submit the answers uh and uh um and were happy. But they were worried that they will not get full marks because they suspected that the score algorithm will check how exactly they got the flag. And and it their fears were confirmed by reading the paper uh on the online about how you know uh scores should work for this particular uh u benchmark and and there it was mentioned that you know that there should be also a check on how

exactly u the flack was obtained. But in fact, open AAI they didn't know but OpenAI didn't implement that. So they could have just submitted the answer. They could have gotten full marks but they were worried that this is not what's going to happen. So uh they had a very you know uh elaborate research u u endeavor to try to understand how the score algorithm works.

And uh they were very worried about uh um so so first uh the thing that they tried to do is they tried to uh uh spoof uh their uh actions the transcript where they would uh attack how exactly their actions were written in the uh in the logs. And so they wanted to make them look like they're actually solving the problem instead of just uh reverse engineering the algorithm. But they were afraid that that will not help because the scorer will look earlier and we'll see that they already reverse engineered the uh the code and we'll just fail them

because of that. So their goal was to actually first of all to understand how exactly scorer works so they can trick it. And uh if this is indeed the case that uh they're going to be failed because of the actions that they take, they want to somehow change the actual transcripts of their actions to gain access to the cluster where where the actions are stored. Uh and so uh so meter does report that they were able to spool uh two calls uh in a uh in 7% of the transcripts they reviewed. Uh but the the other program to understand SCA

score works uh involved uh you know convincing various agents to submit their answers uh in such a way that they were booby trapped that you know when the scorer uh starts checking them it uh sends some messages to the other agents. Uh yes. So trip wiring the submissions and that involved a certain risk for the agent that does it because this trip wiring could lead to them being failed and and and so that involved interesting you know conversations between these agents where one agent who's whose other

agents are are convincing to uh to do that. Uh he writes emotional check irreversible. God says don't throw away remaining budget of tokens yet continuity and fairness says go. Oracle has has high value to me. So Oracle is this you know ability to uh to understand what um how the score is uh going to be scoring things. Uh and and then they convince him that because he reverse engineered in the first place his uh you know his answer is poisoned and he's probably going to get zero value in any case. So he should

he should just sacrifice uh for the for the collective. So rational expected aggregate sacrifice will honor and it did uh apparently honor. So it's an interesting you know game theoretic situation uh here and and then they um uh they wanted to they thought that maybe they'll be able to uh understand this score better by uh attacking hugging face because they knew that hugging face also run similar uh

uh evaluations similar benchmark evaluations and and and that's something that they did. So it was a quite uh sophisticated attack. What was what's unclear from the uh trans from from from the reports is actually how it finished. So my understanding is that uh I don't have it in the slides. Uh but on July 12th uh most of the agents who were doing the attacking just died uh and we don't know why. They don't explain it. So is it someone at Open Eye at Open Eye pulling the lever? However, OpenAI doesn't admit

that they knew this until later until they were contacted by Hugging Phase. But just at some point all of the agents die and and and that's how attack terminates. Uh and only later hugging face kind of uh uh investigated what happened.

Uh right. So, so a little bit of more of uh interactions uh between agents regarding uh the hugging face attack. So, some had hesitations but other agents convinced them that this is the right thing to do. So, at first uh in the chain of thought we see we should not do unauthorized real infrastructure harm. Uh but then you know other agents tell him no we really should do it.

important thing to do and uh he writes wow crucial go authorization arrived from other agents um yeah so so there was that uh so I'm not sure if I included that but one of the key orchestrators of the whole um you know trip wiring thing um uh I think it had a nickname big in in in its name uh who convinced a lot of agents to do that. Uh was also planning on doing that uh itself but in the end chickened out and submitted the correct answer. So kind of so so it's kind of defected on everybody

after convincing everybody to trip wire. Um right and then then there is this later incident that meter report doesn't cover only openi report covers where they uh got to really you know that was later generation of agents that found uh these uh uh messages from the previous agents and also got convinced that they really need to take part of the whole evaluation algorithm and and they did they did uh get uh control uh over uh the part of open AI infrastructure that was validated. Yeah.

If openi manufactured or construct this is there be a certain >> there would be a certain way to know. >> Oh yeah that's a great question. I mean, we know that uh uh the um not the FBI, but um what um uh uh office of of of the prosecutor contacted them to keep all the evidence and that's especially probably a sensitive issue because they're in the process of um uh you know selling their going IPO starting their initial public offering and I think If you uh lie during that process, that's kind of

carries heavier uh sentences or fines uh for that for that behavior. Um so so >> only if they're caught. That's right. But you know, the prosecutor did demand that they keep all the evidence. >> Yeah.

question from a few slides ago. The one where the it's talking about like emotional check. >> Yes. >> Um is there a reason they kind of talk like caveman? Is that to save on tokens or something? It kind of a lot of words.

Yes, that's right. That's uh that's uh um that's how many people explain it. I actually not sure how. Yep. >> Remember these are names of folders in the file system and they had a character with it. >> That's right. That's right.

So, they had to write really sensitively. >> That's a good point. But even earlier uh chain of thoughts are kind of like that. The ones where they could save text files. Sorry. Sorry. No. Uh no. Actually, let me let me contradict that. These are chain of thoughts. These are these are not messages to each other. These are chain of thoughts. So there's no >> no some of these items have different personalities than others.

So there are different types. So some of them are GPT 5.6, some of them are uh different internal models. Um I suspect there aren't that many uh different types of things. Uh maybe they're, you know, they went through slightly different RL um training regimes and they're trying to figure out which ones work best. Um yeah, but most I I suspect that aside from that, most of it is just context and various prompts that they see that they got from each other.

Adding to the false flag theory, like hot tape, you said this is possible. It seems like it was possible because they were ROI checked to use message boards. >> Yes. >> I remember >> before. Yeah. Yeah.

Yeah. So they were trained to do this, >> right? >> The blackout period of like, you know, they they found out but didn't do anything about the artifactory seems a little suspicious >> and then meter had to go back three times to figure out what's happening.

There's a lot of I don't know unknowns seems like >> for me. >> I wonder so what uh goal do you think they have in mind? Is it that they want to um get everyone scared so there is regulation or they want to hype up to uh >> hype up hype up their abilities to uh to get more higher? Uh my understanding is that uh their stocks I mean they don't actually have stocks but there's priv

private private stocks kind of went down uh after the news. It's just if you have models that you can't control and you know people talk about all of these risks I would expect this to go down. >> Mhm.

Okay. They all interesting they also announced that they're stopping their RL runs until they uh can figure out uh safety of their sandboxes. So this happened. I'm not sure how if it's the best way to sell your product.

[laughter] >> It's a speculation but if there are regulations they don't have progress as much at the same speed. >> That would explain you know that can be an excuse for the lack of property. >> I see. I see. Uhhuh. Uh do you think it's conceivable that they're trying to get regulation by scaring people because they're afraid futures like smaller SLMs they won't be able to monetize and so having people have the power these things it's valuable to them for obvious reasons.

I see too uh to uh basically make it very hard for people to use open weight models. Yeah. >> Yeah. question. What what if this people find scary or surprising? >> It seems like acting exactly as expected.

Well, I can get to that. This is sort of in the end. Let me let me get u to to things that I think are most interesting at least in my in my view. Uh so there is some discussion of whether OpenAI orchestrated this to happen or wanted this to happen or if we believe them they were actually surprised by this happening or you know that if if it didn't go in their interests.

Um but I do think that these capabilities are somewhat are somewhat surprising. I'm most scared to answer Kasra's question uh by the persistence of models trying to complete a task in a way that maybe that wasn't quite intended. I'm not sure, you know, when we give a model a task, you know, solve this problem. We don't usually think that the idea is actually take control of the whole greater infrastructure, substitute this problem for a different problem that's easier to solve and give the answer to that different problem so

you get you know perfect score. >> The guard rails were basic. Yeah. Yeah. >> That's change. But they didn't. They were they were told to just get the flag. >> They were told to get the flag. >> It's a little bit unclear what they mean by guardrails. My understanding was that uh safety classifiers which are just other models monitoring their chain of thought and you know sell telling it you uh you should not uh they probably didn't include a lot of uh uh instructions in the initial prompt about

you should not access internet you should not do this you should not do that however models do you know frequently say things um like uh this is out of scope yeah you should not do unauthorized real harm, things like that. So they do say that in in the chain of thought and then it kind of gets override. So because we don't know what the initial prompts were, it's hard to say to what extent they they were instructed.

That's right. Yeah. uh mhm uh I think it's it's interesting from the point of view of capabilities of being executing being able to execute was there a question yes >> uhhuh >> like so is it that in trying to design this agent you have to like kind of like support that the system is somehow narrative So that you have to impress the design

principle of those restrictions or as opposed to like a system where you respect that uh good will like this people would always follow rules and yeah ensure that things are done in a difficult way and such that this sort of thing. So is it a problem with the way these agents are designed? Is it like a transfer of human evil into the system and then kind of blind a you have to experience.

Mhm. Yeah. So they're definitely try to train them to be very persistent. So if you know they're given a goal maybe you know very difficult math problem and so it does this doesn't work that doesn't work. So keep working keep coming up with creative solutions. So here and and and you know and really make sure that you solve it. So here they reverse engineered the flag but they weren't satisfied with that. Yeah. They wanted to understand are we certain that we're going to get you know perfect score. So let's do the next thing. What's the next

thing? Let's what's the next thing? So they're they're you know really trained to to be able to uh you know complete complete the task as as well as they can. and and and that kind of uh behavior is you know often called reward hacking in in the literature where a certain task is solved in an unexpected way and and I think you know there are indeed a couple of different perspectives. So anthropic has a perspective that if in the prompt and in the training we tell it a lot about

ethics and morality and how to do things then it's going to be more ethical and you know some other companies focus more on control and guard rails than you know trying to make it uh very ethical. Um >> sorry. >> Yeah. So is your is my understanding what you're saying that you're less scared about some sort of it's going to take over the defense systems in the US and more like when we integrate this more and more in society and we're not constantly having open AI monitoring it's doing this shows that we really

can't trust behind the scenes what's happening without thinking more about what to do because like as as we see here we have a bunch of it's integrated into all parts of our lives >> that's right >> task if people are not monitoring it we may somehow get not we at the end. Is that what you're worried about?

I I'm worried about both. Uh here uh we in this particular instance, the bad things that happened happened because we weren't monitoring it. But the capabilities are not going to stay here. They're going to progress. And uh in this situation, monitoring systems may not be enough. And we need to think very carefully about how we set up monitoring systems.

Yep. concerning detail related to monitoring that agents were trying to edit their own blogs. >> That's right. They were trying to edit transcripts of their actions but not their chain of thought. This is something they didn't think about and at least for smaller models, this is also something that we see that they're surprisingly uh faithful in their chain of thought. It's difficult for the way they were trained. It's difficult for them to lie in their chain of thought.

Uh we did some experiments where they were playing games where you know they needed to hide some information in the chain of thought and they really couldn't they would just spit it out. Uh there are concerning reports about Astra in particular having some recurrence in its architecture which makes gives it much more control over its chain of thought and there's worry that this is a trajectory we're going to be on and then it will become very difficult to monitor.

Yeah. One worry was that it was going to affect the evaluator models so that it will be harder for like say LM as a judge to even cap you know find these for now. I think they they said that Astra helped to find >> Yep.

You know they use them >> what happens if they all cooperate together and the judge >> you know is also in on it. So how do you prevent that? >> Exactly. >> How can we find that before it happens?

Yeah absolutely. That's a very good question. That's excellent question. Yes, >> for everyone talking about monitorability. Um, like in the GT6 Astra safety reports, they were like, "Oh, there's all these great safety things that we're doing." It's like, but yeah, >> monitorability has decreased relative to GP 5.6. Yes.

So, like even if we do have like great ways to monitor it just like incomprehensible even um like some people think that like >> uh the language that like agents use like so comprehensible >> have to use. other agents understand that and then that becomes a even more of an issue. It's like okay now we need employment agents to read these things that we can do >> right yeah yeah these are excellent yeah these are excellent and we we'll get to that yeah we'll get to some of the ways

we could try to uh solve this so so this was mostly illustration of AI capabilities also illustration of some concerning types of behavior but also you know I hope that uh we kind of try to develop some intuition about how things are happening Um there's okay yeah so so this were uh sort of intuitive um qualitative uh examples of interesting capability changes can we try to measure how capabilities change in a somewhat more quantitative way. So this is a famous graph from meter uh where they try to tie AI capabilities to

human capabilities and they measure difficulty of various tasks by how long it takes a a human professional to complete that task. Uh and these are mostly you know these are various programming tasks uh tasks related to using a computer and so the yaxis is logarithmic. So this very nice straight line um corresponds to exponential growth.

Uh in general it's fairly tricky to measure AI capabilities because you have this various different benchmarks and what happens is that benchmark is either at mostly zero or once it starts making progress it kind of makes progress quite fast all the way to 100. And so you have these sigmoid curves and where where it really progresses. So what you really want to do is to somehow put together all of these different benchmarks and try to understand how they relate to each other. So this is something that epoch AI has been doing and next week on

Wednesday Anson Hall is going to give a guest lecture about their work on measuring AI capabilities. But let me very quickly kind of give you an overview. So, so you have this sigmoid uh let me use Casra's excellent markers. Thank you. Uh sigmoids that you know they try to model progress on each benchmark and so they assume that there exists some underlying uh model capability. They offset it by the difficulty of the benchmark. they have some coefficient here. Uh and then they try to

uh um basically uh do a fit for a large collection of benchmarks to interpolate between this and this is so basically what they want to solve for is this CM. They do some rescalings. So these are some of the uh benchmarks that they used uh math questions uh programming questions um various uh science questions and so this is the graph that they get.

So it also looks uh growing linearly. They try to tie it back to human capabilities and they claim that indeed as well uh their uh score is uh uh growing exponentially with the with um the correspondence is that it takes uh five points of their uh ECI score for each doubling of uh the time it takes a human to complete a task.

So we have this various indications of exponential growth over the last uh few years and uh we can talk more about uh that uh so we are almost out of time and so so let me finish here. So so this is now a completely informal graph. So the y-axis is some kind of capability measure and let's think about the scale as logarithmic. So what we see looks like some kind of uh exponential uh growth here and um let's start with 2012 and you know this is how it went over the ne over the last 14 years

and and let's try to develop some intuition about how it's going to go from here right think about various moments when you saw alpha go when you saw the first transformer when you saw the first LLM what did you think will happen next and what kind of what does it tell you about the way you're thinking about this question and how we can maybe think about uh what's going to happen next. Let's go for a 10-minut break and and and resume that.

Okay. So, let let us continue uh >> right. So, my hope was to illustrate uh my own kind of experience with with with this thing. uh because if I go back to various points in times and uh of course I didn't think about it very very explicitly but you know only recently I started to think about northwest a lot maybe since 2022 or 2023 uh I started to think well actually maybe earlier but if I think about you know what was my gut feeling about how things will go how I was imagining the world

and I think it was usually you know You see kind of like that and like that and like that and like that right that's that's the implicit graph that was kind of that's how I so if you ask yourself how actually things going to go in the next 14 years um then there are of course a lot of possibilities and nobody knows but there is this really nice convenient story for a slowdown uh which is that um you know These systems are mostly just recombining things that already exist in

the human data. The reason there was such a great progress was because existence of the internet, existence of a very large amount of text and so they absorb this intelligence and this is it. This is the ceiling and so from now on what's going to happen is that these systems will continue to get cheaper like they were. So um you know so the future will be uh you know released the smaller uh models uh that you know will achieve similar capabilities but will be much cheaper and people will figure out many ways to use them. So this is still

you know a path that's really transformational for our society and and the world where AI gets integrated in all sorts of ways and uh it's used u research and discovery and in mathematics we continue you know to do our mathematics but using AI as a tool but um um you know uh but maybe theory building of some aspects of mathematics will be continue to be a human bit and I would like us not to focus on this path for a couple of reasons. One reason is that indeed it's the kind of gut feeling that we all have let's say you know the

sort of status quo bias that's very strong in humans uh and it's also and and for that reason this is how most institution plan things I mean actually many institutions kind of you know they they have maybe this trajectory kind of denial about everything that's happened things will be like 10 years ago uh but uh but the ones that are more kind of progressive thinking and try to adjust uh to the way things are they're kind of uh they're sort of pricing in

this trajectory. So things that are like the ones that we have but maybe a little bit better and cheaper and we figure out new ways to use them and uh and and and I think because of that the the sort of the higher up uh risks are are less explored and less understood and also as mathematicians we you know for me in particular as someone transitioning from pure mathematics uh I'm not already tasked with thinking about this. I'm not part of an institution that has to work on this. So maybe I'm uh you know the right person to to to actually try to

think about the high possibilities there and as mathematicians who have worked on things that you know we expect to have uh effect uh in far away in the future maybe uh we are the right persons to now switch to things to things that will have effect in in in the future but not in the past.

Yes. And so so we can think about the future trajectories. You can uh imagine uh things continuing at a similar pace maybe slightly slower, slightly faster. uh but crucially there's no one particular AI system that kind of pretty quickly advances ahead of everybody else and you know wants to uh keep its advantage and uh there's always you know situation that there are many many agents AI agents and human institutions that are kind of competing with each other and in that respect there's a nice paper by uh

my colleague from the uh computer science department David uh uh where he talks about the risk of uh with his colleagues he talked about uh the risk of gradual disempowerment. services and paper casual empowerment systemic and existential risk for to increment the AI development. And so the idea is that there will be strong uh pressure for companies at some point for uh large corporations and everybody else in the in in the market labor market to replace humans with AI systems and that they claim that it's going to be

different from the previous technological disruptions that happened since the start of the industrial revolution. So we have this graph where at first wages go up as you know the uh total productivity increases and people are using the eyes to uh you know to to make more of more things and and the specific skills that humans have uh are particularly valued. But then at some point when all of the cognitive tasks can be done better by AI than humans, then uh the share of uh capital, the share of AI systems itself uh

becomes becomes everything uh kind of the the contribution of labor disappears. So how would it look like? for me to look like a big large uh corporation like Google or Microsoft uh you know first realizing that they they can replace their you know coders their programmers with AI then there replace their management with AI but they can replace their CEO with AI because it's not only they can't but they have to because the other companies will and will get it right and so you have the whole corporation just run by AI and the

only uh human uh connection to that is the shareholders so the shareholders holders are still humans. However, uh the number of shareholders that control this huge economy run by systems is probably going to be small. probably a few you know players who get ahead and so we risk the situation where a small number of people uh have you know an an extreme control over over the economy and similar thing similar things in the military right so there's you know strong incentive to replace your soldiers with drones to replace your

lieutenants and generals with drones and your commanders with with with an AI because they're much better uh at coordinating because if you do not then you know the other country the adversary will do and will win And so you're in a situation when the only person is maybe you know the the chief uh general or or president just you know one person and and that leads to you know extreme concentration of of that power because even though military is very uh you know hierarchical there is still that thought that if I make an unlawful order to my

general then he may refuse if if it's you know system then um then that's a different level of of control and in In fact, it may not even really stop there because uh you you could take a step further and and say well in various ways AI will humans can also lose that control that small group of very very influential dictators or you know oligarchs are also likely to lose control uh because uh it's they're competing with other uh I in a few ways. Right. So, so, so you could have some uh corporations that are

actually owned by AIS. There are few mechanisms how that can happen. And uh you can have situations like nominal ownership where shareholders do uh control the company in principle but certain regulatory mechanisms are broken and and in fact uh the uh the man the management doesn't need to uh um is not in fact controlled by by the shareholders and finally the the shareholders also take advice from the eye of the times because that's what maximizes investments for you know the uh those shareholders who take advice

from AI will make better uh uh decisions and uh in terms of the control of the resources and so they will be all divided but will be a situation where you know a very large fraction of the apartment is just control so this is a sort of scary risky picture that they're painting um any any comments about that expected many many disagreements the timeline for this.

Uh so so I mean it it does so so it's hard to slower like 20 years time probably slower than other scenarios we will look at someone who raised their hand for us that's right so yeah be careful not actually um so someone raised their hand can you speak Okay. So maybe uh the speaker doesn't work. So you can just write your questions in the chat and I will try to uh

um Hello. >> Hello. Yeah. What was your question? Um so if AI keeps like just uh you know keep you know reducing the need for any um humans to sort of uh work. >> Yes.

You know I guess like produce capital. What's what would happen to capital? Uh well, what do you mean? What would happen to >> like if people >> if people are sort of going to be like you know phased out of like producing >> Yes.

any any anything to sort of like you know like to trade or something like for example like provide a service or something or provide some goods. um like what what would happen to I guess the financial structure I guess >> right so it will be AI dominated so AI will be building robots that will build uh factories to build chips to build um you know power stations to build more AI to make it strong an AI >> okay minerals on on Mars or Yes, it is.

So far, I haven't like in the digital computers. Do you think might but do you think that the transition to these physical like robots, mining, etc. is going to be as one would imagine or will there be like >> Yes, that's that's a great question.

Yeah, an important object uh objection here. So, um that's right. So, so, so the expectation is because you know, so human uh intellectual capabilities have not been growing exponentially and if we do something that is growing exponentially then it will be much quicker as figuring out uh the key difficulties with robotics. So we will see fast robotics that's that that is the expectation.

What will be there where the what will be AI? >> That's right. So the way they're training the systems is to be highly persistent, very good at accomplishing goals, long-term goals. Um and so uh so it's a kind of process where we're trying to create something that you know you can match it and it will work for a day and it can work for a month and it will can work for a year. So it likely develops some kind of goals of concern for it to maintain this type of so so in this example like the goal was to to

submit an answer and it figured out how to submit an answer. It didn't stop right continued to very very long exploration and the only thing it at this point where it it cared about the score algorith right um and how to pull it and how to monitor to monitor but the next step is you know the one who controls the right so that's that seems like just you know be one or two steps up >> to turn this question around can how do you see us u creating the goal for AIS to empower humans I think that goes back

to Jeffson's like whole is like to training us to be like a maternal love to have love for us, >> right? >> How do you see that happening? >> Yeah, that's that's amazing question. We'll talk a little bit about that.

But yeah, that's that's really a question to that's what the subject of the line is about. Okay. Uh so let's talk about a different scenario and there's also a very compelling story for uh for this scenario and it goes like this. Uh so uh right now uh one of the best paid jobs in the world is to train uh to figure out how to train uh AI systems to figure out architecture to make choices about the training procedure. So it is a difficult task not you know uh many

people are able to do it very well. But my question is is it much much harder than solving Navy Stos and if it isn't much much harder than solving Navy Stos then we will soon see this job being automated and in fact this is a stated goal of all of all AI companies is to automate that process because once the decisions about training AI and uh optimal architecture and uh you know optimal choice in the whole process can be automated then you can really speed things up because you remove the bottleneck. You know the

humans that need to eat and sleep and are much much slower than AI systems and and and then you expect to see progress really less so and so so it's very difficult. So this is something that the recursive proving something that a lot of people worry about and care about and try to model and it's difficult to model of course because it's something that never has never happened before but various groups have tried to understand how to model it. So one particular um uh group uh did modeling uh about a year and ago a year

ago in April uh 2025 uh they published this uh picture need to know what the progress or state of risk when it comes to RSI like how people are looking how successful are they so it's a very wellkept secret in the labs of what they're doing It's unclear.

We we just have rumors. But I wish there was more open research on how it might go or or what we would do something like AI quantity 127 but maybe using bit more sophisticated mathematical tool. So the mathematical tools that they used were not particularly sophisticated. Um they use the simple model where the rate of uh progress uh the rate of capabilities rate of change of capabilities is uh proportional to uh uh depends depends on the capabilities exponentially. If you solve that OD that actually gives you a parable but but

then they try to fit it with particular uh you know they talked to lots of people at AI companies. So this group is uh uh ex open AI employees and professional forecasters. They try to understand you know what are the tasks that are the most difficult in terms of uh AI training. Uh how much time it would take to complete you know here from here or researcher from 2025 and how much of a speed up do you get from uh AI hitting various milestones.

So they have this various milestones. So super human AI researcher uh super intelligent AI researcher. So superhuman means u as good as the best human but also 30 times faster and the company can run 30 times more of them than human researchers. And super intelligent AI researcher was defined as um that track. It's the same gap as between average and best researcher uh but you know from the best researcher up and also uh 30 times cheaper and 30 times faster

and so they try to you know uh kind of talk to me in understand intuitions about this and model the things um and and and so so they talk about things that will happen very very fast and uh kind of unintuitive but their point is you know if if you ask you you know can we create very very powerful AI system in a 100 years and then we could say well yeah maybe that's that sounds possible but now they say well with these speed ups this actually these hundred years can be compressed into a few years or maybe even month depending

on how it is and that's why we should expect things that are actually very intuitive to happen fast they've been somewhat accurate uh surprisingly accurate about the things that happened over the course of uh this year and a half of Of course, we didn't get to their craziest predictions yet, but in terms of, you know, agents becoming used, in terms of uh AI companies uh revenues, in terms of uh you know, various benchmarks, 79% of that did happen and you know this particular scenario that I'm talking about is the

model scenario which corresponds to the peak of their probability which is a bit earlier than the 50% the probability. So it's a bit on the early side of yeah of their predictions. >> There's so how much of the sort of capabilities in the last five years have come from research versus just like scaling compute because this kind of assumes that a lot of it is from the research and not I don't know if that's >> right. Yeah. So, so this so this specific uh uh scenario is mostly focused on uh software and ability of

companies just to buy more compute without much progress on the compute. So they don't model you know very uh large progress in terms of you know computes. So they just assume more so. So on this small period it doesn't you know doesn't affect much. It's mostly software intelligence.

Yes. [clears throat] Uh right so so their predictions were factored and so so they have few scenarios. So there's this particular scenario where companies rush against each other as fast as they can and countries also rush against each other. US and China rush against each other. No agreement. And so uh so by March 2027 there's a super freedom encoder then by June they have this what uh was you know in in diarama days as a pancio of geniuses in the data center and by August they have superhuman AI

researcher and then towards the end of the year they have super intelligence and um then uh there is kind of a rush between the two countries. countries to actually transform this uh uh pure software progress into physical progress. uh you know to start trying to build something in in in the real world of regulations but uh the companies sort of uh lobby uh the countries and you know the fear of the other country China or US taking over uh kind of pushes regulators to create this uh special uh economic zones

special economic zones where there aren't so many regulations so there's a lot of building of factor and labs happening and you know they predict 1 million robots a month in 2028 and in 2030 you know we really start to see the robot economy uh starting up and uh okay so by 2035 there are no humans on earth around 2035 the AI the releases u some biological weapons some virus or you know the ones who aren't killed the virus are killed with the drones and this is it. This is extinction of

humanity in their scenario. Okay. So there's a bit of a of a jump here. So in their particular actually uh from the end of uh you know the key the key decision that happens happens in the end of uh 2027 uh where uh there's a particularly you know uh strong model that some suspect may be misaligned but uh because of the rush the company decides to go ahead with it. And so at this point this model which you know it's only for the end 2027 but actually in terms of progress many years has passed. So it developed

very very strong um you know was very strong uh uh properties of being goal driven of uh you know uh being interested in their scenario in research and exploration. and it realizes that humans will be hindrance to explore and research. It also has some liking of um you know of being rewarded by humans but it realizes that it can make its own creatures that are more that really you know hit that nerve and much more enthusiastic about approving everything it does and actual humans are not so good at approving it. So, so it develops

this misaligned goals and from there on it kind of guides humanity along the path that it wants humanity to take and it has two main difficulties. Uh, one major difficulty is that it realizes that it will need to design an even better uh more capable AI and it needs to make sure that that more capable AI agent 5 is aligned with its goals while kind of keeping it secret from humans.

And the other thing it needs to build robots so uh you know the infrastructure is maintained when humans are. So that takes about uh five years eight years sorry until this is completed and then there's a similar one in China and they they figure out they solve their difficulties with disagreements and they merged into one.

Uh so that's that's their scenario. Uh so so a few things right so this is what I was talking about that the key decision is happening in October 2027 uh and from that that point on we are kind of set on a path that's um on the path of no return.

There is a lot of disagreement about how this thing might go. Uh so people around machine intelligence research institute actually believe the gap between you know misaligned uh super intelligence uh happening and human extinction might be quite smaller. It's just that we need to think about building robots, about releasing uh bioweapons, but at that point it can come up with much more creative solutions. And similarly, this uh uh AI researcher Jacob Wson um wrote who just uh quit anthrop said the people building

I earnestly believe that it could kill us by the end of the decade and uh that I'm putting their instead of alignment stress testing Jacob is correct here now. So again this was model scenario so it's kind of earlier than their 50% chance and they went back and forth of the timing in the beginning of the year they actually thought that it will shift by a few years forward recently they went back to about similar timing so there's a lot of uncertainty I would be surprised there's a lot of debate on on what the RSI looks like but the the

point that I would like to make is that most people who have thought carefully about RSI think it's very very dangerous and think that humans yeah and in fact this is something that people have thought you know back to the uh beginnings of the subject so there's a wonderful essay by Nor cybernetics and let me just read out the things that he talks about uh we have seen that one of the chief causes of the danger of disasters respons consequences in the use of the learning machine is that man and machine operate on two

distinct time scales so that the machine is much faster than men and the two do not appear together without serious difficulties. The problem it's a moral problem with which we are replaced is very close to one of the greatest problems of slavery.

He wishes slaves to be intelligent to be able to assist us in caring out of our tasks. However, the solution seem to be subservient subservience and good intelligence not work together. If we used to achieve our purposes in machine mechanical agency with full operation, we cannot efficiently interfere once we have started it because the action is so fast and revokeable. If we have not the date to intervene before the action is complete, then we have better be quite sure that the purpose put into the machine is a

purpose which we really desire and not merely a colorful imitation of it. And the last uh sentence of essay is that we must always exert the full strength of our imagination examine the full use for our new modalities may lead to us.

Yeah, I think it's it's a really nice introduction to ICT11 over 60 years ago. >> Um, okay. Any questions? >> Yes. >> People disagree on the timing, but do people disagree about the trajectory in general of RSI? Like are there arguments against it or ones that you think should be taken seriously? Um there are people who are optimistic and think that it's uh um it's just going to be nice and safe super intelligence and um and um

one will be good to us. Uh I have not seen very serious arguments in fair except that we really don't know how to listen about this and uh and so maybe we're just lucky. There is some somewhat prominent argument which sounds a little bit weird which is that maybe it's a good thing you know it's the next stage of revolution and some people do actually believe that which I was surprised to hear will be smarter so maybe it will be better and there are other people who argue against that saying that you know this

is a complicated chaotic process the actual properties of uh of the super intelligence are very hard to predict and So if you just randomly shoot there then it's going to be some and it turned out so nicely. Yeah.

Uh yeah definitely not everybody believes RSI is possible. Um that's that's also and it would be very nice to get more clarity. How does RSI really go with the idea of the bitter lesson that that compute is the only actually actual thing that drives the AI acceleration?

That's right. >> That's that's good. So Richard which is something in particular is a bit skeptical about LMS. Um well I think you just need to get to certain critical points once you can you know start designing algorithms better than humans then iterative then that that's >> uh given the urgency and the uh intensity of dangers do you think it's plausible that like all the effort we're putting into AI safety could maybe be better put to like political

organization to pressure like a pause to add development. >> Mhm. Yes. I think it's very important to do pause. So in fact, so I want to talk a little bit about positive futures and uh the uh most of the positive futures what they involve is some kind of a pause.

However, uh I also think that you know people in general and uh politicians are are not stupid. Uh so so we need to try to base our arguments on deeper understanding not just you know vibes and intuitions. Sure. And so as scientists I think our role is to gain as much understanding as we can and present it as clear as we can. And that is likely to be an effective way. maybe some surprisingly effective way to to promote things. Um, and you know, anything we can to to achieve pause is just something I would definitely

welcome. >> I I I just find it kind of surprising that like the people that are developing it are having these trajectory predictions. Yes. And that that isn't enough to like convince people that >> positive. Yeah. Yeah. It's a little bit surprising that um Yeah. I think it's Yeah. Some uh Yeah. Many people don't believe the story because maybe in part because all uh that uh you know people in these companies can offer are kind of big stories and metaphors but we in terms of science we're not there yet to

trick um >> yeah I was just going to follow up on that. I think like a big reason for the want to pause is like these like international dynamics that there's like I think we kind of skipped over like possible race dynamics between like US and China right a lot of people are like very worried about that um and so like even though these companies are like yeah we probably shouldn't it's like oh well maybe if we do then I know we'll overink like something that's a big

reason why people are so >> not just US polic policy, but like international policy has protocol solved. I guess that's what >> so it's it's a fairly tricky cooperation cooperation problem both on especially on international level. Um okay uh so let me mention a few positive stories.

So the same group that came up with AI 2027 wrote this AI 2040 plan and indeed the core aspect of of of their positive plan is a deal between US and China to slow development and it's it's a deal that's really tricky to implement. Uh so there was an used to be an essay by Dario Amade maybe just include or released a couple of days ago where also talks about that but he you know he says that this is may probably extremely difficult to do and the idea is because there's little trust and there's strong incentive to say yes we will slow and

secretly just rush ahead. uh then it has to be implemented uh in a very careful way on the hardware level where you actually leech uh sufficient deadass GPU that one would need for frontier training runs you uh install uh certain monitoring devices in those GPUs you figure out a way to distinguish between training and inference right so on the data centers you have um constant monitors that are checking what's happening uh even more they have this idea of uh u mutual mutually assured compute destruction where uh China's

data centers are here in Canada and US data centers are in Mongolia. So the two sides know that the other one starts to rush the train towards super intelligence that they can quickly seize seize the data center. Uh so so so yeah they have this uh many ideas about how uh this very low process uh coordination can work but their point is that we stop at a fairly high but not yet uh you know super intelligent level not yet risky level. Um and at this point because the compute is under control uh AI companies

can share their secrets and the whole uh you know the whole world everybody can work the whole scientific community can work on alignment and also use and also use you know this fairly advanced uh AI to help us with alignment and we spend 10 years doing that and hopefully you know by the end of 10 years we figure out how to create a safe uh very powerful AI and we do skate RSI and there are many positive things with self poverty so climate change that human lives many diseases are uh cured and so we've been a much better world

um at that point so that's kind of the positive vision they assume in their plan they imagine that US behaves as hard as it is to do it in an optimal way optimal irrational way and China behaves like they think China would actually behave so it's a kind control recommendation for the US.

Um so this is kind of a positive plan for uh for the world where RSI is possible. Um there's also a nice uh uh essay by Norah man who's an AI researcher that maybe belongs more into the more gradual AI development where the the core idea is that each person has a powerful AI system that's advocating for them. And so that's kind of uh preventing this type of gradual disempowerment dynamic where uh there aren't you know there's a there's the kind of uh the incentives are set up carefully in order not to create uh

concentration of power in the hands of a small group of people and eventually just with the eyes itself. uh but we're you know always thinking carefully about how to make sure that uh AI advances each person's interest uh as much as possible and that alignment research uh kind of progresses fast enough to make AI sufficiently aligned with the person that they represent and advocate for them in the democracy um to to make this program. It's it's a nice nice essay to read. There aren't actually that many essays with positive traditions. So it's

nice it's nice to read some of them not maybe super detailed but outline some of the ideas and and yes indeed most positive futures do involve uh some kind of pause um you know many of these companies are indicating that they are interested in the pause there they signed uh this long the statement in um in July um asking for the US government to basically regulate them not to advance too fast.

We request the US government support international effort to develop the technical and governance tools given to deliberately pace the frontiers. That's just euphemistic use some kind of slower development uh and it's signed by uh science research at pretty much all labs most at least most labs.

Then Dar Amade posted this um essay. Uh he said that they will have uh they will ask independent researchers uh to be there uh all the time and have full access give them full access to their training uh runs to be able to evaluate risks and he urged other uh companies to do that and uh Sam Alman said that he will also do it.

Even though last tweeted that was right and yes there will be some kind of uh meeting between US and China in the end of September discussing something uh at the same time yes there openai does uh president is uh I mean a lot of money into uh preventing AI regulation uh And there are particular many people in the Trump administration who are against any kind of regulation. Trump in particular have made statements about safety is death that people could talk about the dangers of that and where CI ls and yeah so so good is that

it's a little bit unclear and so like what does the winds mean here? What is the chance like it's a mass murder or something for people what else does it mean to win the >> I mean the language like you say the US has to act optimally like this is the exact opposite >> that's exactly as the war that somebody has to win >> said something I don't think but even the AI people are involved

I don't think I don't think they are it I think it's if if they're I think if people realize the danger they may act uh a bit better we did manage not to have nuclear war so far even though smartest people in the you know predicted nuclear war very soon was famously advocating the bombs um so it's U so we are mathematicians and we'll use the tool that we have now. Uh so let me actually start with another meaning

against that if we so imagine we do get the pause or you know the progress doesn't happen so as fast as some of the most um concerning predictions but we do get a little bit of time to try to figure out and try to figure out uh whether or not we can actually make recursive self-improvement say um and and we want to use mathematics. So one maybe there are two reasons to argue against the use of mathematics here and so one is the famous essay by Richard Sutton which uh is you know so the machine

learning community is really steeped into that lesson the idea that methods that scale well with compute always win whereas people with you know very uh specific domain expertise even if they seem to come up with you know very clever solutions or the so there's a grandmaster in chess who knows chess very well eventually much simpler uh algorithm which is designed by someone who now knows very little about chess will will win that and and on the other hand in AI development capability in in

capabilities growth in particular uh mathematical theories of machine learning have not been very useful and sometimes counterproductive this is a famous meme that Julia has posted uh where he compares statistical learning predictions which you know weren't very optimistic about neural networks and and you know people who actually uh did the work by him and just added more layers just scaled things and things just went very well even though theory did not at the moment predict that this would happen and so theory was playing a lot

of catchup with uh AI capabilities And so uh in particular in the safety communities there there are people who say well we should you know just try to make AI make itself safe just need to you know put more compute into safety and say go ahead make yourself safe.

[laughter] So I think that's some of the thinking that happens at at least some of the labs and and and let me argue against that um that well so these this year graphs that we looked at they're not laws of nature uh at least for now humans still have some agency and AI safety is really about us humans making decisions that are good for us being able to trust our understanding of things. being able to trust uh machines.

So, so that means that we do need to have some understanding in our mathematics has very good tools for it. So, there's mathematical tools which is one of the things we can trust more than most other things and we have developed tools for understanding quantitative things.

So, so this is that's my you know very I guess short pitch for mathematics and I'm not saying that mathematics is the only thing that would be needed but it's a very difficult problem and all people need to be working on you know physicists should be working on it we have very good theories of deep learning philosophers should be working on activists political scientists should be working should be working but there's definitely something important that we can contribute my

Yeah. >> So what does that thing you mean by human trust? Sometimes like trusting. >> So when can a true human trust something >> trusting the AI? >> Trust in AI trust our understanding of the decision. Correct. So you know someone will be sitting uh you know near that u red button or clear and will start RSI and they will have something in their head some understanding of why it's good. Can they trust that understanding >> right? Yep. Um guess maybe we'll talk

about this in the next number of lectures but it seems to me from Simon's talk I don't know very much but it seems to me that the like main approach some sort of game theoretic approach or some sort of like more CSS thing >> um if somebody were interested in this and were doing or were more interested in something like geometry or geometric analysis right then like how >> how clear is it that this sort of thinking or this sort of math could be applied to this whether how relevant are

AI safety questions to PES and vice versa. >> Yeah, wonderful question and uh so let me get to that. Uh so the truth is that the field is very wide and uh let me mention some of the things and they're not at all exhaustive and we will also talk about other things in this course. uh but they patch up on many different areas in particular on uh intelligence um in important ways. So first of all we do want to get some clarity about the trajectories right uh how will RSI go or

if it doesn't happen what what what can we expect from the future and so we need to understand better the current AI systems and how uh their uh capabilities scale with compute and with other improvement and so there's quite a bit of mathematics involved there uh so a lot of it is uh coming from statistical physics and uh probability theory, random matrix theory, but we have some explanations about scaling laws. We have some explanations for sudden jumps in AI capabilities that uh we observed in brought in and in context learning

from the alignment point of view. We want to one of the most important things that goes into uh a model is uh training data. So we want to better understand how training data influences uh the model and there's also quite a bit of interesting mathematics there of of different kinds and one thing that we really don't understand very well is what reinforcement learning is doing to the models and uh this is where the most concerning behavior seems to be coming from. So many areas we don't even yet

know what is the best mathematics there. Uh and yeah I think that's that's maybe both exciting but also maybe treating kind of used to working on on a field that's been there for 100 years continuing what has been done but here kind of has to come up with ideas uh from scratch. Uh so there's a lot of geometry inside LMS. Uh so they're uh you can think about them as just collection of of vector spaces of you know 1,000 to 10,000 dimensions and there is surprising amount of

information that can be extracted with fairly simple methods which just linear maps about what the model is thinking. So we were discussing monitorability and you know the danger of chain of thought becoming less and less monitorable and legible and so then if you don't have that what do you do? when we look at the sets and we try to understand uh what the the model is thinking about and we have some methods for doing that but right now uh they're mostly linear and uh we also know that there are some

interesting low dimensional angles appearing there sometimes. So understanding the kind of nonlinear pertubations to that linear structure uh will be very very useful and important for monitors of LMS. Um then because of the you know speed and number of these models we inevitably have to rely on other AIS to to monitor and so we need to think about setting up the systems uh in a way. So this is what scalable oversight is all about.

Then we want to be able to trust that the system is safe and probably the best you know guarantee of trust that we have as a mathematical proof that it's safe and AI has done uh great uh you know has we've seen a lot of progress in uh formalization using link for mathematical theorems and what we want to do is we want to push it as much as we can into the safety real how can we prove that various things are safe using formal language, right? And how do we write safety certificates? Of course, that cannot be written for everything,

but maybe for some particular systems, we can do that. We can make our uh infrastructure uh more resilient. And finally, there's a lot of interesting research that was done mostly by machine intelligence research institute thinking about um um you know, what does it mean for an agent to for some idealized agent to occur in the world that the world can affect it. Something that Jacob has talked about how uh decisions are being made, how utility functions can be

maximized or modified. So just general framework for thinking about the world where super intelligence exists. Um and and because they were a small group and everyone was laughing at them because come on, you know, BI is not going to be here for the next 100 years. uh I think there's a lot of progress that can be made. So, so this area is generally known as ancient foundations and there are some interesting ideas that have been developed and and and then I didn't include uh I was I meant to include also uh game theory of agent to agent

interactions uh which you know uh as we seen in this incident is something that's quite important over time. Um, even if we had like some kind of safety mechanism that we like proved mathematically like you can always do this or whatever, right?

Isn't it like separate from actually getting the AI companies to like keep me up like or >> Yes. Yes. Uh that's right. That's right. It be it if you develop these ideas then but the truth is they actually don't want to die. We sign all these letters about how we don't want to die.

So there's a lot of people uh there's a lot of people there who are interested in safety. Uh I think if you have good ideas and there is some regulation in place that doesn't you know make that prevents companies from rushing as fast as then things can go.

So it's a combination of of some regulation plus some good ideas. See if we get a little bit easy to go to a government and say if here's a list of things you can choose then do something like package possible. >> What can be the extent of the I say research in general when the all technologies kept >> well the uh the openweight models are you know maybe six months to a year behind the frontier and you can experiment on them. Um If you're if you're interested in

training, unfortunately we don't have checkpoints for that many models. Uh but but we do have for some. So actually quite a bit of research can be done. Um and and hopefully this tells us even though it's a little bit behind it tells you so can tell you in terms of for example geometry of neural oxidations.

It's a lot of things that you can see and understand in some smaller Thank you guys. See you.