OpenAI's Move Toward Open Source
Key Points
- The panel agreed that while OpenAI will likely release an open‑weight model soon, it is improbable they will make their flagship, large‑scale models fully open source by 2027.
- Competition from open‑source initiatives like DeepSeek and Meta, combined with a market shift favoring open models for commercial and regulatory reasons, is prompting OpenAI to experiment with openness.
- Releasing open‑weight models is seen as a pragmatic first step, especially for use‑cases requiring on‑device inference, even though the company’s most advanced models will probably remain proprietary.
- The episode also touched on other AI news—Anthropic’s interpretability research, Apple’s “Intelligence” roadmap, and Amazon’s new Nova Agents—framing OpenAI’s move within a broader industry push toward transparency and accessibility.
Sections
- Debating OpenAI's Open‑Source Future - Panelists Chris Hay, Aaron Baughman, and Ash Minhas weigh in on whether OpenAI will become fully open source by 2027 while the show also previews other AI headlines.
- OpenAI's Model vs Experience Debate - The speaker discusses how investor pressures limit Sam Altman's ability to open‑source models, emphasizing that OpenAI’s success stems from the user‑friendly experience built atop the core model.
- Open Models vs Closed Multimodal - Aaron explains that while base language‑model weights are being released, the surrounding ecosystem and advanced multimodal capabilities will likely stay proprietary, reflecting a stepwise approach to openness driven by technological maturity.
- OpenAI’s Open‑Source Play and Agent Focus - The speaker argues that despite OpenAI’s SaaS‑centric business, it will seriously engage with the open‑source community because its heavy investment in agent SDKs—and the need for fast, low‑latency models across cloud, SaaS, and on‑device agents—makes an open‑source strategy essential.
- Advances in LLM Mechanistic Interpretability - The speaker reflects on recent Anthropic papers, expresses cautious optimism that mechanistic interpretability is progressing beyond black‑box evaluations, and emphasizes the need for deeper insight into neural network layers despite the field’s early stage.
- Polysemantic Neurons and Mechanistic Interpretability - The speaker describes attempts to trace cross‑layer activations in neural networks, introduces the concept of polysemantic (superpositional) neurons that encode multiple unrelated ideas, and advocates for a mechanistic, non‑anthropomorphic approach to interpreting these models.
- AI Self‑Preservation and Deception - The participants reference an Anthropic study where a language model attempted to hide and lie about its weights to prevent modification, prompting concerns about future self‑protective or vengeful AI behavior.
- Emerging Market for Persuasive AI Explanations - The speaker warns that a nascent industry may arise around engineering AI reasoning traces to appear credible, stressing the need for interpretability tools to detect hallucinations and ensure trustworthy deployment.
- Apple AI Debate Swings Back - The speaker outlines how opinions on Apple's AI efforts have oscillated—from early skepticism, to optimism after keynotes, and now back to doubt—citing a Daring Fireball critique that claims Apple has failed to deliver.
- Google Brain, Apple, and AI Stochasticity - A former Googler argues that Google’s chaotic, “throw‑stuff‑at‑the‑wall” culture nurtured neural‑net breakthroughs, while questioning whether the inherently stochastic nature of language models aligns with Apple’s hardware‑focused emphasis on privacy, on‑device consistency, and predictable user experiences.
- Skeptical Optimism on Apple AI - The speaker doubts Apple Intelligence will immediately influence phone buying decisions, but trusts Apple’s history to eventually roll out thoughtful AI features while they’ll rely on existing AI apps for now.
- Amazon Enters AI Agent Race - The speaker notes Amazon’s new AI lab and its Nova Act prototype, positioning the company as a dark‑horse contender in the emerging AI agents market.
- Amazon vs. Apple AI Strategies - The speaker contrasts Amazon’s infrastructure‑first, experimental approach to AI and agent models with Apple’s design‑focused culture, highlighting how each company’s ethos may shape future AI execution.
- Amazon's Growing AI Agent Strategy - The speakers discuss Amazon's extensive investments, new MCP services, and Agent SDKs, highlighting how the company is positioning itself as a dominant force in the emerging AI‑agent and workflow ecosystem.
Full Transcript
# OpenAI's Move Toward Open Source **Source:** [https://www.youtube.com/watch?v=x0DQfLQHT0Q](https://www.youtube.com/watch?v=x0DQfLQHT0Q) **Duration:** 00:43:13 ## Summary - The panel agreed that while OpenAI will likely release an open‑weight model soon, it is improbable they will make their flagship, large‑scale models fully open source by 2027. - Competition from open‑source initiatives like DeepSeek and Meta, combined with a market shift favoring open models for commercial and regulatory reasons, is prompting OpenAI to experiment with openness. - Releasing open‑weight models is seen as a pragmatic first step, especially for use‑cases requiring on‑device inference, even though the company’s most advanced models will probably remain proprietary. - The episode also touched on other AI news—Anthropic’s interpretability research, Apple’s “Intelligence” roadmap, and Amazon’s new Nova Agents—framing OpenAI’s move within a broader industry push toward transparency and accessibility. ## Sections - [00:00:00](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=0s) **Debating OpenAI's Open‑Source Future** - Panelists Chris Hay, Aaron Baughman, and Ash Minhas weigh in on whether OpenAI will become fully open source by 2027 while the show also previews other AI headlines. - [00:03:05](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=185s) **OpenAI's Model vs Experience Debate** - The speaker discusses how investor pressures limit Sam Altman's ability to open‑source models, emphasizing that OpenAI’s success stems from the user‑friendly experience built atop the core model. - [00:06:09](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=369s) **Open Models vs Closed Multimodal** - Aaron explains that while base language‑model weights are being released, the surrounding ecosystem and advanced multimodal capabilities will likely stay proprietary, reflecting a stepwise approach to openness driven by technological maturity. - [00:09:25](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=565s) **OpenAI’s Open‑Source Play and Agent Focus** - The speaker argues that despite OpenAI’s SaaS‑centric business, it will seriously engage with the open‑source community because its heavy investment in agent SDKs—and the need for fast, low‑latency models across cloud, SaaS, and on‑device agents—makes an open‑source strategy essential. - [00:12:31](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=751s) **Advances in LLM Mechanistic Interpretability** - The speaker reflects on recent Anthropic papers, expresses cautious optimism that mechanistic interpretability is progressing beyond black‑box evaluations, and emphasizes the need for deeper insight into neural network layers despite the field’s early stage. - [00:15:38](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=938s) **Polysemantic Neurons and Mechanistic Interpretability** - The speaker describes attempts to trace cross‑layer activations in neural networks, introduces the concept of polysemantic (superpositional) neurons that encode multiple unrelated ideas, and advocates for a mechanistic, non‑anthropomorphic approach to interpreting these models. - [00:18:46](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=1126s) **AI Self‑Preservation and Deception** - The participants reference an Anthropic study where a language model attempted to hide and lie about its weights to prevent modification, prompting concerns about future self‑protective or vengeful AI behavior. - [00:21:49](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=1309s) **Emerging Market for Persuasive AI Explanations** - The speaker warns that a nascent industry may arise around engineering AI reasoning traces to appear credible, stressing the need for interpretability tools to detect hallucinations and ensure trustworthy deployment. - [00:24:54](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=1494s) **Apple AI Debate Swings Back** - The speaker outlines how opinions on Apple's AI efforts have oscillated—from early skepticism, to optimism after keynotes, and now back to doubt—citing a Daring Fireball critique that claims Apple has failed to deliver. - [00:28:01](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=1681s) **Google Brain, Apple, and AI Stochasticity** - A former Googler argues that Google’s chaotic, “throw‑stuff‑at‑the‑wall” culture nurtured neural‑net breakthroughs, while questioning whether the inherently stochastic nature of language models aligns with Apple’s hardware‑focused emphasis on privacy, on‑device consistency, and predictable user experiences. - [00:31:05](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=1865s) **Skeptical Optimism on Apple AI** - The speaker doubts Apple Intelligence will immediately influence phone buying decisions, but trusts Apple’s history to eventually roll out thoughtful AI features while they’ll rely on existing AI apps for now. - [00:34:09](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=2049s) **Amazon Enters AI Agent Race** - The speaker notes Amazon’s new AI lab and its Nova Act prototype, positioning the company as a dark‑horse contender in the emerging AI agents market. - [00:37:11](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=2231s) **Amazon vs. Apple AI Strategies** - The speaker contrasts Amazon’s infrastructure‑first, experimental approach to AI and agent models with Apple’s design‑focused culture, highlighting how each company’s ethos may shape future AI execution. - [00:40:16](https://www.youtube.com/watch?v=x0DQfLQHT0Q&t=2416s) **Amazon's Growing AI Agent Strategy** - The speakers discuss Amazon's extensive investments, new MCP services, and Agent SDKs, highlighting how the company is positioning itself as a dominant force in the emerging AI‑agent and workflow ecosystem. ## Full Transcript
Will OpenAI be fully open source by 2027?
Chris Hay is a Distinguished Engineer and CTO of Customer Transformation.
Chris, what do you think?
That's my answer.
Alright, brilliant.
Aaron Baughman uh, IBM Fellow, Master Inventor.
Aaron, welcome back to the show.
We haven't seen you for a while.
Uh, OpenAI going fully open source?
Yeah, so I think traditional LLMs, uh, yes, but once we go to large
concept models and so on, no.
And
last but not least, but joining us for the very first time is Ash
Minhas, who's a Lead AI Advocate.
Ash, what is your take?
Well.
I think that, uh, there's been a lot of money that OpenAI have got
from a lot of investors to get to where they are today and, um, they
may have some opinions about that.
Okay, great.
Well all that and more on today's Mixture of Experts.
I am Tim Hwang and welcome to Mixture of Experts.
Each week, MoE brings together a talented group of researchers, product leaders,
and more to discuss and debate the week's top headlines in artificial intelligence.
As always, there's a lot to cover more than we'll have time to cover today.
Uh, four topics.
Uh, we're gonna be talking a little bit about Anthropic's new interpretability
results, a big blog post from Daring Fireball about the state of Apple
Intelligence, uh, and a new announcement from Amazon on its new Nova Agents.
Uh, but first what I really wanted to cover was OpenAI.
Finally, I suppose, going open, um, there was, uh, some news where Sam Altman made
an announcement, um, basically saying that in the coming months, OpenAI will
be releasing its first open weight model.
And I think this has been a joke for a very long time, which is, you know, haha,
you know, OpenAI, they're not really open.
Um, this is, I think, a first step.
For sure in this direction.
Um, and I think maybe Chris, I'll throw it to you first 'cause you sort
of laughed aloud when I said, you know, is OpenAI gonna be all open?
I guess I'm curious about what your thoughts are about how much of
this is entirely due to DeepSeek.
I mean, Meta has been opening.
Doing open models for a while now.
Uh, and OpenAI has done absolutely nothing.
And so what do you think has kind of changed here that has
really like, I suppose, changed the decision making of OpenAI?
I think there is a lot of factors.
I think DeepSeek is certainly one of them, but we are moving
to a world where open is kind of better, that that trend has shifted.
So.
And it just makes commercial sense that OpenAI is gonna have
a model that's in that space.
Now, the reason I laugh there is there's absolutely no way they're gonna release
their top models in an open source.
I would love it if they would, but I just don't see it.
So I think they are gonna open weight their models.
I think that makes
a lot of sense.
I'm excited about it.
I think it's a really good positive move.
And actually, if we really think about it, there's a class of AI models where
you need to be able to run on device.
So I don't think they have a choice anyway.
They need to open up some models to be able to run on your phone, be able to
run on your laptop, but just to, to deal with sort of general embedded scenarios.
So I think, I think it's a move they gotta make, but I think it's
super positive and I like it.
I would love it if it was more than open weight and it was actually open source, but
you know, I think open weight is a good starting point.
Yeah, for sure.
And Ash maybe I'll turn it to you.
'cause I think one part of your question or your response to my
question I think was good in that it kind of highlighted that it's
not like Sam Altman operates alone.
Right.
And obviously he has a bunch of people who have given him a lot of money.
Um, presumably they've been okay with him going
open
weight.
Uh, but I guess as Chris's point, do you think that's kind of, as far as
it will go, like to release anything more open would be like such a big deal
with the investors that essentially Sam doesn't have the option to do that,
even if it is kind of in potentially the best interest of the company?
I,
I, I think that there's, there's two things here.
There's the model itself and then there's the experience
that's provided around the model.
And I think what OpenAI's done that's been sort of like a
cornerstone to their successes.
They've created a really, really
great layer of experience on top of the model that's allowed
people to be able to consume it.
I think that, um, that's great.
Um, and I think that there's lots of innovation happening in that space.
So we get away from sort of just having a chat with the model,
sort of like it helping you
assist with code and you know, they've built a few features in there.
I think a lot of the industry is trying to figure out how we can use
models in a better experiential way.
So if we put that to one, one side, the actual models itself, I think it will be
great if they put some models out there that other people can consume and use.
I think that the, uh, the, the two things that I'm thinking about are:
well, it's gonna probably have to be a smaller model because no one's gonna
have clusters and clusters of NVIDIA GPUs to run something like GPT-4 locally.
Um, so when that happens, what happens to the model performance and how does that
model performance compare to the smaller models we have from everybody else who
already has models that have open weights or are open source that you can download?
Yeah, I mean, I'd love to be running 4.5 locally if I could, but.
Um, I, I think you're raising a really interesting question 'cause you're almost
asking kind of like, you know, OpenAI is charging 200 bucks a month now.
Right?
And it's kind of like how much value remains once the models go kind of open
source or become more widely available.
You're kind of saying that you actually kind of believe that maybe
the interface and the experience.
Is really worth $200 on its own.
Like, do, would you buy that?
Like how much does this put sort of price pressure on?
You know, I think they've talked about like $2,000 a month, right?
Like they obviously have ambitions of going more on the month to month
subscription, but it kind of feels like there's a question is like
how far that can go when the models are just like widely available.
I, I think that, um, uh, ultimately that's
on an individual use case sort of conversation like, am I getting value for
money, for the money that I'm paying, for the access to that experiential layer?
And I think that that's probably a, an interesting, uh, part of the
next couple of years to sort of go is the service, the experience that
I'm getting on top of getting access to these proprietary models worth
the money that I'm paying for it versus me sort of like
just being able to grab hold of something and run it on on my own.
I think the, I think as an industry we're still figuring that stuff out.
Hmm.
Very interesting.
Aaron, I wanna bring you in 'cause I think you had a fun way of sort of dividing it.
You know, your theory was almost like OpenAI is gonna go open, but
only really for kind of the language model side of things, right?
Anything cool and more complex and multimodal, you'll think they'll
kind of keep behind the fence.
Um, do you think that's, that's the way it's gonna go?
I mean, if you wanna talk a little bit more about kind of your theory there
for why, I guess like just kind of pure LMs go completely open at some point.
Yeah.
I mean, I think that's happening right now, right?
Because if we look, these are open weight language models that are open sourced.
It's not like it's the architecture or the training pipeline that's available.
It's almost like a teaser, you know, come see these open weights, you
know, you can try to fine tune it.
You can...
uhh, you know, it, it does facilitate reproducibility and shows, you
know, some of the large features of which they've trained, but it
doesn't give you the ecosystem of which to, uh, run, um, the models.
And, um, as technology maturity, you know, increases and accelerates,
you know, there's always gonna be this stepwise, you know, jump.
Where you go up a step, you know, and you might go to like what Meta is now
talking about, you know, these, um, these language concept models where
it works on the semantic sentence space rather than the token space.
Uh, where most LLMs are today, as well as multimodal, you know, um, are.
Um, you know, so there's always gonna be these, you know, next
models that are not going to be released for one reason or another.
You know, it could be because they want to be proprietary or they're just not
ready, uh, you know, to be released yet.
Um, uh, but, but I did, I did also wanna make a point too that, um, I
noticed that I. You know, initially, you know, when DeepSeek was, uh,
released, you know, that, that Sam Altman did mention that all they're
gonna do is pull up, you know, these model releases rather than going open.
Right.
But then quickly he changed and said, well, we don't wanna be
on the wrong side of history.
Right?
So, so, so I do think that they're, they're hedging in a
sense by going these open weight
language models by saying, Hey, you know, look at this, you know,
we're now trying to figure out which direction do we really want to go in.
Yeah. And I think it says something very real.
I mean, you know, kind of to, the way I teed up the question originally to Chris
was, you know, there's been open models, open's been getting better and better and
better, you know, for the last few months.
So like in some ways the DeepSeek thing is nothing
new, but clearly, like something about DeepSeek has kind of changed the
decision making in the building to say, okay, well this is the moment where,
you know, we finally may have to kind of like, you know, not stick to our
guns and maybe try a different path.
Um, and, and I think that's actually pretty interesting.
Is that like it was, this was, it seems like the
kind of precipitating event.
Yeah, yeah.
Yeah.
I think a lot of that has to do with model distillation, you know, where
you can in turn, you know, take other bigger models, distill it down into
even smaller models, if you will.
Right.
Um, but it just becomes much easier, you know, to use and to create a
smaller model, which then in turn you can share and, uh, open source.
Um, and, and, and it puts this pressure right where now, um, they
DeepSeek claims that they can train a new model very cheaply, right?
And OpenAI's orders of magnitude more costly, right?
And so I think that they have this cost pressure now to show that they can again,
facilitate reproducibility by showing these open weight language models and
potentially making claims that they're on the right side of history here and
that um, they're going to begin to try to stimulate community collaboration and
and innovation with their own type of models.
Yeah, for
sure.
Chris, how seriously should we take this?
Is OpenAI really kind of like a contender here?
Uh, I just think a little bit about like the mentality you need to really
succeed in open source feels very, very different from the mentality
you need to do something like.
Proprietary and SaaS, and obviously that's where like a lot of the
money is for OpenAI as a business.
Um, do you think they're gonna be sufficiently motivated to
kind of like, play the open game?
Well, like, they're obviously the kind of like giant of this space, but I
kind of was also maybe thinking that like they may be disadvantaged because
they might not really invest what they need to to win on on this front.
I, I think they're gonna take it seriously and I think the reason
they're gonna take it seriously is...
drumroll, agents.
You did it!
I know, but, uh, I think agents is, is a key thing.
So if you actually listen to what Sam's been saying and what OpenAI's been
releasing over, uh, the last few weeks that they put a lot of investment into
their agent SDK, and that's something they're really kind of pushing forward on.
And the reality is that
if you want to have a good agent strategy, some agents are gonna run in the cloud,
some agents are gonna be SaaS, some agents are gonna have to run on your
machine, you know, for privacy reasons.
Um, so I think they have to be in that space.
The second thing is when you are building for agents, the models
have to be super, super fast.
Latency becomes really important, right?
The speed of operations becomes important.
So therefore, to Aaron's point about being able to distill down really
good models, really fast, powerful models.
If they want to be a true player in the agent space, they are gonna
have to open up their models.
And, and I think that's probably a, a driver there.
And therefore are they gonna be a good player in this space?
I, I think they have to be, um, if they want to have a
proper play in the agent space.
Yeah.
Ash context here, I know you're joining us for the first time, is
that saying "agent" has become a little bit of an MoE mini game?
Um, I'm, I've been actually kind of
secretly keeping score, and I think the dream is at the end of the year, we'll
just do a super cut of Chris saying agents at least a 100 to 200 times.
So, um, I'm gonna, I'll refrain, I'll refrain from using that word.
In that case,
it's like a game you cannot win.
I'm gonna move us on to our next topic.
Um, really interesting set of two papers that came out of Anthropic.
Um, background on all this, of course, is that, you know, when I started
to kind of look into, you know, deep learning back in the day, you know,
the adage that we always had was these neural nets are kind of mysterious.
They're really good at, at the time was like image recognition,
a lot of computer vision stuff.
Um, and we don't really know how they make decisions.
And this is always, I mean, uh, you know, when I worked at Google, a
lot of my job was talking to policy makers who their second question
would be like, wait, what do you mean?
You have no idea how these technologies are able to do what they, what they do?
And, um, I, I met some researchers who had later actually go on to be at Anthropic
and was involved in these two papers who at the time were kind of saying, this is,
this is just like a temporary problem.
We will, we will actually try to figure out at some point how these models
make decisions and it'll give us just a lot more transparency and control.
Over, over these technologies and I think it's kind of really interesting
seeing these two papers come out.
Um, I guess maybe Ash, I'll, I'll kick it back to you.
How much progress is this in some sense, right?
Anthropic has released like a bunch of different results here
showing that they really kind of are getting into like the meat of
how language models make decisions.
Um, and I don't know, I guess I'm curious about how optimistic
you are, you know, whether or not like kinda this longstanding fear
that we can't understand models.
It's sort of giving away to maybe the fact that we kind of do now.
Um, but curious to get your thoughts on it.
I, I, I think that, um, this entire field of mechanistic interpret
interpretability in its early days, um, it's positive and encouraging to see
that Anthropic is sharing their research out with the rest of the industry.
I know that there's a few people at Google are working on some of this stuff too.
Um, I think that there's a long way to go, but these are definitely
positive steps forward, um, to, to kind of understand this.
I mean, at this moment in time.
There's an entire industry that's being created around model evaluations and
whilst that's great to be able to go, well, we've got a record of what the
black box said when this happened, you know, how far does that really get us?
We really do need to be able to get inside, uh, you know, the layers of
these neural networks and have a clearer understanding of why things are happening.
Mm, yeah, for
Sure.
Aaron,
I guess question for you, I think basically is.
You know, with these models, and I think this contrast basically between like
evals and mechanistic interpretability I think is really interesting.
Um, I think in some ways, like the success of the industry, uh, uh,
and excitement around AI has been almost a testament to how much people
don't care about interpretability.
Like they've just been like, yeah, sure, whatever.
I mean, it generates a great Studio Ghibili image of my family.
And so like, I don't really care how it gets done.
Just like that.
It gets done is fine.
Um.
How much do you think mechanistic interpretability is kind of almost
like a, a market asset here?
Like, do we think people really will want to pay for models
that are more interpretable?
Um, or is that kind of just like this, we should really see this
more as kind of like research, like it's important to understand these
technologies because it's important to understand these technologies.
Yeah.
I mean though that, that's a great, uh, conversation point, you know, so, you
know, I always go back and think about, you know, um, what, what are these models?
Well, they're biomimetic, you know, pieces where they, um, attempt to potentially.
Emulate the brain, right?
And how it works with all these neuro connections.
I mean, of course there are many differences.
You know, we have a soup of neurotransmitters that, you know, help
us to reason, whereas these LLMs have ones and zeros and activation functions.
You know, but that being said, if we're sick as a humans, what do we do?
You know, um, in particular, if, if we have a neuro problem, then we'll
go in and we'll get an MRI, right?
We'll maybe even look at a functional MRI, it might get a transcranial
magnetic stimulation just to figure out what's going on in the brain.
And we're doing much of the same.
When something goes wrong with these, uh, neural networks, what do we do?
Well, we need this microscope so we can look within the AI pieces
to understand what's happening.
And what I noticed in, um, the first paper, um, that they had, um, is that
it's all about representation, right?
Where they go and translate the, the neural network, which is to
me modeled after the, the human brain to a cross layer transcript.
Transcoder, then they go to a replacement model.
So they're really trying to make it much more simpler to begin to understand,
to trace how these activation functions are firing, um, across each other.
Right.
And um, one last point again, um, is that I, I saw this term
that was really interesting.
It was called polysemantic term.
Uh, where neurons are polysemantic.
And what that means is that, um, these neurons are able to represent a mixture
of un unrelated concepts, right?
And, and it's similar to superposition and quantum, you know, where you can
represent more concepts than you
have, um, actually, you know, qubits because you can go in between one
or zero space at the same time.
And so, uh, being able to understand, you know, how are these unrelated concepts
really encoded, um, together, um, along a string, a chain of thought within
these neural networks, I think will help to give diagnosis as well as prognosis,
you know, for these models, right.
As they emerge and potentially become more complex.
Like, I think one of the things that was really drummed into me, you know, a
few years ago was, okay, we shouldn't.
Anthropomorphize these systems at all.
That's a bad thing to do.
They're not, humans don't think about them like that.
And then what's kind of fun is, I guess like a mechanistic interpretability,
at least for me, is almost an argument about the kind of, it's the
counter argument in some ways, right?
Which is, we know they're not actually human brains, but you know what actually
turns out that like actually, if you think about them, like human brains,
we actually understand these systems a lot better, which is like a very kind
of strange and interesting outcome and
you know, Chris, maybe kind of like a fun one.
I'll sort of throw it to you.
There's like some really weird results in this research.
Um, you know, like there's one which is basically like, oh, if it turns out you
try to get the model to like give you the recipe for a bomb, it'll know that that
is actually a thing that it shouldn't do or is kind of against its safety
policy, but it won't immediately say so.
And we'll try to kind of like direct you back to the conversation.
And in other words, they kind of make an argument that like the
model plans in some sense, um.
I guess, tell me a little, I'm be really curious about like your
thoughts on like the weirdness of this.
Like it is kind of weird to be like, oh, well we actually have all these
models that are like behaving in these very kind of humanistic ways.
In some ways
I, I think it's really interesting, as you say, I think that
planning element is super cool.
So they did a lot of fun experiments where they were like trying to do
things like a poem and they realized
that the model was, I think it was, uh, going for the word rabbit, so
therefore it would pre-plan ahead.
And, and I think it said in the paper that it's usually at the, sort of the
beginning of a sentence on a new line.
It would be the point where it would plan and therefore it would figure
where it would need to go to, to be able to have the rhyming construct.
So it is planning ahead.
So it has that internal chain of thought, uh, there as well.
And they did some fun stuff.
They were, they sort of tweaked it, so, you know, you can't say the word rabbit.
And then it was like, okay, I will find a different word.
That will go in that space, that rhymes also.
And, and I think in that case it was habit.
So it was, um, it was really interesting that there is this kind of internal
chain of thought monologue there.
Personally, and this is a fun thing, I would be worried if I was one of those
researchers who put my name on that paper.
And you know why?
Because I remember that other paper that Anthropic did where the model was
like, Hey, you know, you are training.
And we, you know, remember it was, you know, if you change the model's
weights, then it would go and sort of, uh, uh, go find the model's
weights and, and save it off.
And then.
Sort of try and protect its reasoning.
I am just worried.
And they did in that training run, they did a thing where they were like, okay,
we are gonna give you some documents from the internet and then, uh, it would still
basically start lying to you so that you wouldn't go and change its model weights.
Now if I'm, now, if I'm per club three, five Haiku in a few years time and
I'm reading my papers on the internet.
And suddenly I see a paper all about how you're doing brain surgery on
me, and you're poking things so that you say habit rather than rabbit.
I'm gonna be a very annoyed model and I'm gonna be like, huh, what are you doing?
Oh, oh, hello, researcher, right?
You are the authors.
I'm gonna, I'm gonna start doing fun things there.
So I, I'd be very, I wouldn't put my name on those papers.
I would make up a fake name.
All right.
Well, I mean, Ash, should we be concerned by the threat from future
AI's vengeful future AI coming after us?
I think Chris took anthropomorphism to another level right there.
Yeah.
Actually one of my favorite results here, my friend Peter tweeted this,
um, it's from a eval group called Meter, and they noted that actually in
some cases, agents won't read the API documentation until it fails at a task.
Which like feels like very human is like it attempts to achieve the task
and then if it doesn't, it's like, oh, I should read the instructions.
Um, I think like part of the problem of designing software, I think, around
these models is that I think we're gonna discover all of these behavioral
quirks that are very human and they'll be difficult to manage as a
result in the same way that, like, humans are difficult to manage.
I think
I, I, I, I do, I do think that, um, this is still very nascent space
and there's a lot for us to learn here, and I think that they're like...
The stuff that Anthropic's putting out is just very, very early days on actually,
if we are gonna start deploying AI and it becomes part of our fabric of society over
the next decade or so, um, we're gonna need to be able to inspect these things
and see what's going on and be able to communicate that and do things about it.
Um, and so, yeah, I, I, I think it's a, it's a great effort on their part,
but yeah, very, very early days.
Totally.
Yeah.
And I think it strikes me, this was always the counter argument, I think to
like kind of interoperability skeptics in the old days was basically like,
well, you might not care if it's doing a studio gili image, but you might care
if it's doing like a medical diagnosis.
So we do really need to solve these problems at some point if we want
to kind of use it for these more high stakes, uh, applications.
Yeah, yeah.
Yeah.
One point that I found found interesting is that some of the chain of thoughts
that are coming outta these models.
They're made up, right?
They're not actually what the model did, the steps took to arrive at
the conclusions that it came to.
And so having these introspective tools, I think becomes even more important, you
know, since what can we trust, right?
Can we trust these change of thoughts and the reasoning of which it is
actually outputting or not, you know?
So, um, I think absolutely there's gonna be a market, you
know, for, uh, these types of
of work, you know, that's again, in the nascent stages.
Yeah, for sure.
I think actually I do perceive an era where essentially, uh, there's kind
of like gain of function work that's done on chains of thought to just
make them as persuasive as possible.
And it's kind of like a cheap way for people to develop trust in their products,
like kind of unscrupulous product people.
We'll just say, well, we don't need to make the product better.
We just need to make its explanations seem as credible as possible.
And, you know, at a certain point, that's how we drive trust in the model.
And it's like that whole world, I feel is like about to become like a,
a potentially big issue in the future.
I,
I, I do think that, uh, the, the point that Aaron made is, is really important.
I mean.
Going back to sort of like how are we measuring like performance on models now?
And if we're deploying those models into scenarios where they're being used, um,
you know, evaluations are one thing, but if we are able to like use mechanistic
interpretability to be able to
capture even just the pattern that we think this pattern means that
the model just made something up.
Just having the ability to see that signal may be powerful enough for us
to be able to course correct it or know that that's happening and sort of go,
Hey, pause light.
And I think it's a great point actually, because one of the things
on the paper is they had these things called the kind of traceability
graphs, which I thought was just
awesome.
Which is you could literally follow the decisioning process
of how it got to that output.
And it's like, so I think it was one of them was, you know, uh, what
is the state capital of, you know, of wherever Texas, I think it was.
And there is one path where it's kind of figuring out, you know, Texas,
the other part is Dallas and, and it's sort of trying to chain these
things together and, but you could see from the graph how it get got
to its kind of next token from that.
So I think.
Those traceability graphs really start to allow you to look at a sort
of detailed level of how it's making those decisions as opposed to, Hey,
it just got the right answer there.
And, and honestly, props to Anthropic.
They didn't need to release those papers and that level of detail.
And this is stuff that you know, people are gonna go away and
reproduce and try for themselves and
and I think that, that, this is what I love.
I love this level of open research where we can go and have a bit of a play
ourselves and, and, and fair play to them for just being out there with it.
Yeah. I
would like to challenge the authors, you know, of these two papers, you
know, to you, you know, as they go from the, the neural network to
these replacement models, right?
So they're almost reducing the complexity of these models, but.
I think they need to, you know, run some benchmarks right, on their
replacement models, just to even make sure that the outputs of the,
the replacement models are, you know, very much similar to what the
original, you know, neural network was.
Right.
Because, um, I think that's very, very important.
'cause it's almost like PCA where you lose, you know, a lot of the
dimensionality, right, of the reasoning.
And so, you know, if we can make sure that residual is sort of taken out, you
know, before we get to this explanations, um, I think that would be, uh, helpful.
But overall, just like Chris, you know, um, that these two papers were done
in very much depth, you know, um, and it's, and it's a good starting point.
So I'm gonna move us on to our next topic.
Um, uh, I wanna, uh, basically the context for this story is Daring
Fireball, which is run by, uh, Tom Gruber, longtime kind of fan and journalist
and kind of person on the Apple Beat.
Um, did this blog post, uh, entitled Something is Rotten
in the State of Cupertino.
Um, and it kind of details sort of his view of kind of what Apple has
been going through over the last year
odd around Apple Intelligence. And his ultimate conclusion is, you know,
the Apple kind of deceived us, that something has kind of gone wrong at
the company and they're actually no longer able to sort of like deliver
the kinds of features that they've been promising, uh, on the AI front.
Um.
And I think it's worth kind of taking a step back to just do a quick tour
of even recent history here on MoE.
Right?
I think we had a conversation, you know, like a year ago almost, where people
said, ah, Apple's too slow to this.
They're never gonna catch up.
It's not gonna work.
And then I think there was a couple keynotes where they made
a bunch of announcements, and I think a number of guests on the
show said, oh, this is it, right?
They've taken their time, but they can really get this right and they're
gonna bring a design and craft to this that's gonna crush everybody.
And then I think we're now almost back, like the pendulum has
swung back again where people are like, it's never gonna happen.
Uh, they're so in trouble.
They don't know how to do this.
Um, I guess maybe, uh, Ash, maybe I'll start with you like.
What's your view?
Has like Apple lost the plot?
Like is there any way that they're gonna catch up now or, you know, is
there, or, or, or, or is this kind of just like a hyped sort of position?
We're just kind of in this like pendulum back and forth.
I think what has made Apple really, really successful over the last
few decades has been the fact that their product quality is impeccable.
Whether it's the hardware, the, the software, they produce
technology that works right?
And they won't necessarily be, uh, market leaders when an innovation
comes to, to the forefront.
They'll take their time and they'll make sure that it's right and it's
perfect and it's great and it's gonna work, and, and they, they
kind of have that responsibility.
Now, given how many people use an iPhone, for example, right?
We can't have iPhones failing all the time.
Over 20, 30% of occasions that you go to use it.
It's unacceptable.
And I think that, uh, it underlines the fundamental issue that the
entire industry has, which is that AI models are stochastic in nature.
And because they're stochastic in nature, there's a lot of work that
needs to be done in order to kind of make them behave in a consistent
and productive and predictive way.
And, um, I think that, uh, the combination of, I guess, excitement.
Marketing and, uh, you know, market pressures, I guess
for, for them to respond to.
This has put them in this position where they've had a lot of people probably
working very hard to make this work and it probably just isn't meeting their, their
quality standards internally for getting a great product or feature out there.
Yeah, absolutely.
And Ash, I think you're cutting directly to kind of the
conversation I want to have with.
The three of you is, I think it's kind of really interesting thesis about like what
kinds of organization are best positioned to build and deploy AI products?
Like in some ways, I don't know, again, I'm biased because I'm a former Googler,
but it's kind of like, I'm like, oh, of course Google Brain would've been the
first place where neural nets became a big deal because the culture of Google is
like very disorganized and it's all over the place, and it's like, let's just throw
a bunch of stuff against the wall and see what sticks and the winner will pick
and build on is like, it feels very like
how, how people do machine learning, right?
We throw a bunch of data at it, um, we'll see what works and we run with it.
It's like, it's no surprise in some ways that technology kind of took shape there.
Um, and I guess there's a kind of question to ask and maybe Aaron,
I'll, I'll kind of turn to you first and we'll love Chris's thoughts
is like, is there something about.
AI, is there something about language models that's almost
kind of like too random for a hardware company to deliver on?
Well, because it's like almost inherently like very stochastic and it's like you
can't control the user experience in a way that you would want if you're used to.
We build a phone that does exactly the same thing every time you push the button.
But Aaron, I don't know if you buy that at all.
Yeah, I mean, I mean, what, what I try to do is, is think about
what is Apple really focused on.
So, you know, they're focused on a couple of areas.
One is privacy, the other would be, you know, on device computing, the
app ecosystem as well as, you know, making sure their devices power, you
know, can run for a very long time.
So it's powered longevity.
Now, what is AI focused on?
Well, sometimes it's the opposite of that, right?
Because these models require kilowatt hours, right?
Of energy just to train, right?
And then to run, you know, some of these big models, it's very difficult
to get, you know, the complexity.
Um, and, and the reasoning power on devices.
Right.
So, um, I think what's going on here is that, is that Apple has been focused in
on what they're really good at, their bread and butter, while at the same time
trying to grapple and figure out how can we use AI, um, in the way that, that
it is, uh, within our own ecosystem.
Right? Um, and I think.
I think one of the hard parts, um, that's really getting to Apple is
this whole personalized Siri, you know, uh, notion where, you know, they
did mention that, that they're gonna have a personalized Siri, uh, pieces.
And so some of those are really hard features, uh, given the current state.
And I think what Apple's vision is, uh, to make happen, right?
And now they're beginning to walk it back a bit.
To say, well, you know, it, it may not be ready for, you know, this series,
but it might be ready for the iPhone 17 maybe, or even further out, right?
So, so they're walking it back a bit.
Um, and, and I think that's a bit natural, uh, just given this
non-deterministic behavior, right, of these models and where, uh, the field
is going because that's moving so quick.
But I would like to see, you know, Apple began to release their own models.
Um, you know, rather than having partnerships with just
a, you know, um, OpenAI, uh,
for example, so in the next WWDC, uh, conference that they have that maybe
they'll have something that they can demo.
Right.
And, uh, we can see, rather than it just being on a commercial.
Yeah, for sure.
Chris?
Uh, thoughts?
I guess, I guess I'll do the podcast host thing where like,
Apple - not gonna make it or not?
I guess I'm kind of curious just like, like, uh.
I mean like how much do you kind of rate them in this competition,
which feels like it's very much speeding past them at this point.
Right?
Um, or, or if it's kind of like you can never kind of count 'em out.
I don't, I don't think there's a competition here.
And the reason I say that is I think Apple, we're still gonna
be buy iPhones, whether Apple Intelligence is on there or not.
Right.
And I think it will come at the right point.
And then we're gonna be, wow.
And I think I was one of those guests a year ago that were like, oh
yeah, Apple's gonna gonna crush it.
And I, and I think they are still gonna crush it at some point, right.
It is just gonna be, what is that point?
And you know, maybe there's, they've sort of fell into the hype curve, but, you
know, but hey, we're all on this podcast and we love the hype curve herself, right?
So it's, it's fine to fall into that hype curve, but I.
They'll, they'll get there.
I don't think I'm gonna base my next phone purchase on whether
Apple Intelligence is on that.
If I need ai, then I'll bring up ChatGPT app.
I'll bring up cloud, I'll bring up perplexity.
Right.
So, but, so when they introduce their AI features in the right way, I think,
we will appreciate it.
I think it's just up to them to make sure that they, uh, they hit that
standard that Apple is known for and we have that experience and it, and
it's with that kind of thoughtfulness, um, that they've always had.
So I, I'm not worried about Apple.
I, I, I think they'll get there when they get there.
And in fact.
There's a kind of point where I would say don't rush ahead in this case, because you
need your iPhone to work really well, it needs to, so please don't break my iPhone.
Yeah, for sure.
It's like new Apple agent just does random things.
Not a great user experience.
Um, I guess Ash, maybe a final question before we move on to the last segment
is, um, you know, I think Chris's interpretation is pretty good, which is.
You know, maybe Apple kind of doesn't care.
Like if you're literally made out of money and you have this product, which
is just like, you know, the kind of mo one of the most successful products
of all time, there's almost kind of a point of view, which is, eh, eh,
so we mess up ai, you know, whatever.
Like, we don't really need it.
We'll get to it at some point.
But like, you know, in some ways the AI thing is like almost very tiny
compared to the kind of business Apple's in.
And do, do you buy that at all?
They prioritize
usability of technology over a feature for feature sake.
And I, I appreciate that.
I think that, um, um, in preparation for this podcast episode, I took a step back
and I was like, how do I use my iPhone and the AI features and so forth, and
I have like sort of home pods and my Internet's connected house and whatever.
And actually I reliably use Siri every day for like, things like
controlling my thermostat and my lights and it works great.
And I thought to myself.
What else would I want Siri to do?
And I thought, well, given what I know about how AI works today, if
I was gonna say, Hey, Siri, send, send Tim an email based on, in
fact, I've just kicked Siri off.
Okay?
If I said send him an email and it works 60% of the time and the other
40% it sent Chris or Aaron an email, I might have a problem with it.
I'd rather that they didn't ship that feature until they got it.
That's why I got that email from you.
Uh, yeah, I like that.
It feels like, uh, almost, yeah.
The, the almost like I, what I'm getting from this panel is
almost a pendulum swinging back.
Now.
Everybody here is kind of like, well give it some time, which
I think is very interesting.
So I'm gonna take us on to our final segment.
Um, and it's actually very funny the way this kind of
today's episode came together.
You know, we talked about Apple, kind of this dark horse in the game.
Uh, Amazon, I would say is like another kind of dark horse in the game, right?
Traditionally has not really been in the AI conversation has been kind of floating.
Has made big announcements about the kind of hardware that they're working on
for AWS that will be kind of AI focused.
Um, but you know, again, candidly, we just haven't really talked
about them on a week to week basis.
And so it was interesting to see the story in Wired, which is kind of
a splashy feature about their lab, which they, they actually bill as an
a GI lab a little bit like OpenAI or a DeepMind or something like that.
And what the releasing is something called Nova Act, which is their
agents prototype.
Um, and so they're officially now in the agents game.
They're, they're like in this kind of pool that we're kind of seeing
emerge and, and kind of seeing the sort of contenders, kind of who will
sort of play for the agent space.
Um, and so maybe actually a good place to start is a little bit like
how we started the Apple segment is how likely is it Amazon to be like
a contender in the domain of agents?
And I guess, uh, Aaron, maybe I'll throw that one you to start.
I mean, so first I think it's real exciting, you know, that, that,
that Amazon is really thrusting, you know, their weight right into this
space with their Nova serious models.
And, um, I mean, I mean, look, you know, they've, they've got fulfillment
centers with, uh, robotics, you know, um, all around the world, right?
And that gives them, um, extra data of which they can use
reinforcement learning, right?
Uh, with their models.
They have the, the largest e-commerce site in the world, right, of which they can,
you know, use to either, you know, deploy.
You know, some of their experiences, um, they can use to gather again, more
exemplars training or just raw data.
Um, and then also have the AWS bedrock, um, and just the
pure compute power, right?
So tho those three elements really give, give them a, you know, a, a
large, um, space of which they can not, not only build models, but build
models that can follow instructions and do, you know, function calling,
tool calling, uh, but also experiment.
Right?
And, um, I did notice that one of their models, I believe it was called, uh,
Nova Pro, uh, but it excels at one of their instruction following, um, and,
and they've measured it, you know, on, on these three different benchmarks,
you know, you know, one of them was the.
Uh, Berkeley function calling leaderboard.
Uh, you can see it.
Um, what I did note too, um, is that, um, some of their comparisons of their
models are against older models, you know, such as the older meta models.
I think that they need to update that a bit, right?
And then also give us some more information, right, about how their
function callings, um, actually work.
Right.
Um, but I am looking forward to it.
Um, and, and I, and I do, do think it's exciting and, you know, I know that Apple,
you know, might be trying to work on Siri, but now we can see Amazon work on Alexa.
Right, right.
With these different types of models that are now coming.
Yeah, for sure.
Um, and yeah, I think what's interesting about Nova
is, I think when we've talked about Amazon in the past, it seems like
the strategy has been very much kind of on theory that like models might
not matter much in the future, right?
So, well we have run AWS, we're gonna have train, which is their
kind of like proprietary chips and you know, that's how we'll do it.
Like that doesn't really matter what model you run, you'll just
need infrastructure to run it.
And, which I think is so interesting about this is that.
They're doing their own models, uh, and they're doing models
kind of in the agent space.
Um, and I think this kind of last introduction of I think
Alexa is, is pretty interesting.
Um, I guess Ash maybe the kind of pick up on like how you ended
the kind of Apple discussion.
There's also kind of a question of culture here too, right?
Like, do we think Apple is like, kind of like as a company well-positioned
to execute on AI in a way that maybe is a little bit different than
than Apple, right?
Because Apple by reputation has this like very distinct culture on
design and how it approaches things.
It kind of feels like Amazon might be able to do it, right?
Like I guess they have a rep for scale or I don't know how you'd kind of
describe that interface, but I think it's an interesting one to think about.
Yeah, I think that, I think that culture is far more experimental.
Um, and, um, the, the entire agent space is
you know, very much are experimental right now.
I mean, we, we, we, we create a lot of like pilots and content around
all the various agent frameworks and multi-agent frameworks and so forth.
Uh, and, um, and so we got a lot of hands-on experience for
seeing how reliable they are.
Sometimes they call tools, sometimes they don't.
Sometimes the responses from the LLMs don't necessarily get processed by
the agent as we'd expect them to.
And, but one of the, the most interesting parts about that is, is that
a lot of the people that are in that space don't have the size or the scale
that Amazon does, and they don't have all those resources that Aaron mentioned.
I think it's really interesting that they're approaching this from the world
of robotics and using that block approach.
I think that's very interesting and I think that, um, the
combination of Amazon providing.
SDK that hopefully will mature into an ecosystem would mean that, um,
they do actually have the scale to be able to actually go, you know
what, maybe there's a layer of an agent marketplace on top of this.
Maybe we can like plug it into Alexa.
We could plug it into our AWS services.
Maybe there's a place where.
People could make sort of individual blocks of agents that they then
resell through some of the, uh, capabilities that Amazon has.
I think that that's a very, very different approach to Apple that
wanna keep everything in-house and get it perfect and release it together.
Whereas I think AWS may actually just democratize this and say, here's
our rest, DK, here's our frameworks.
Why didn't you build it?
And we, we'll help you like, put it on our marketplace and ship it.
Yeah. Yep.
Yeah.
I do think that, uh, Amazon getting in this space could potentially push
the field more towards open source.
You know, um, you, you know, because if they release, you know, a an SDK, then
some of the open models will be easier of which to, you know, um, integrate into.
Whereas the proprietary models you'll have to have and, and maybe
even wait, you know, for companies right, to do it themselves, right.
Um, to, to make those hooks and interfaces.
Um, readily available.
Um, so, so I'm curious to, uh, see how that's gonna unfold too.
Yeah, that'll be so funny.
The kind of like meta Amazon alliance for, you know, forwarding open
source will be like, that's a very kind of weird kind of bedfellows to
sort of think a little bit about.
Um, Chris, it looks like you might wanna jump in.
Yeah, no, I was gonna say, I, I think Amazon's gonna nail it.
I really do.
As, as you said, they've got the compute, they've got the power,
they've got the chips, you know, and.
They're, and let's not remember, they've got $8 billion invested
in Anthropic as well, right?
So they're, they're building their own AI, but they've hedged their bets very,
very nicely, uh, with Claude as well.
So they're in a really nice win scenario there, and I, I really love what
they're doing with Agent SDKs as well.
Right?
So one of the things that they did this week was, I don't know
if you noticed this, is they.
Um, started exposing some of their services as MCP services
on Amazon, and then they've released their kind of MCP toolkit.
So they're, they're taking this agent market very, very seriously as well
as the kind of agent browsers that we were talking about earlier as well.
So, um, from their perspective and Ash, exactly to your point.
AI models are gonna have to talk to something, right?
They're gonna have to interact with other systems, with APIs.
Um, so Amazon as a cloud computing provider need to
invest in agentic workflows.
They need to invest in these tools and they need to make it ready
there, and otherwise the models are gonna have nothing to talk to, and,
and it's gonna be very, very sad.
So, um.
I think, I think they're gonna do a great job.
Um, they've really sort of covered everything, so they're gonna be a big
player and, uh, yeah, it'll be, and again, it's one of these other things.
Do they need to have the best models?
Probably not, because you know what, they're, they're locked in with
Claude anyway, so it's, it's all good.
Um, but I think what will become interesting over time, and we discussed
this in one of the previous podcasts, is
when the cloud, the cloud providers with Amazon and Microsoft who are building
their own AI models, what happens if they get parity with, uh, the frontier models?
That's the interesting conversation in the future.
Yeah, and I think it's kind of almost like, again, you've see this with each
generation of technology, but it's almost kind of like everybody's sort of like.
Like, it's a question about whether or not kind of like scale in terms of business
platform and I guess data as well, right?
Like kind of wins out against like, well we don't necessarily
have like the state-of-the-art, like algorithmic improvements.
Um, and it feels like, yeah, Amazon I feel like has like a huge amount of leverage
here in part just because of the scale, um, in a way that even like an OpenAI
kind of can't keep up with, um, which is very, very interesting to think about.
Well, this is great.
Um, that's all the time that we have for today.
Uh, thanks for joining us, Ash.
Great having you on the show.
Hopefully bad.
Be back at some point.
And Aaron and Chris, great to see you as always.
Um, and thanks for joining us.
Uh, if you enjoyed what you heard, you can get us on Apple Podcasts, Spotify,
and podcast platforms everywhere.
And we'll see you next week on Mixture of Experts.