MIT vs Wharton AI Success Metrics
Key Points
- Conflicting AI ROI studies (MIT’s 95 % failure rate vs. Wharton’s 75 % success rate) are creating widespread confusion for businesses.
- MIT’s unusually strict success criteria require measurable bottom‑line financial impact within a short timeframe, inflating the failure rate.
- Wharton relied on executive surveys that include broader metrics such as productivity, time savings, and throughput, yielding a higher reported success rate.
- The disparity between the studies is essentially an “apples‑to‑oranges” comparison, not a direct contradiction.
- While MIT’s high bar emphasizes the need for rigorous financial validation, a balanced approach that also considers operational benefits would give enterprises clearer, more actionable guidance.
Sections
- Decoding Conflicting AI Success Rates - The speaker critiques wildly differing MIT (95% failure) and Wharton (75% success) enterprise AI studies, explains how methodological filters create contradictory headlines, and aims to give businesses a clear, realistic understanding of AI project outcomes.
- Balancing AI ROI Perspectives - The speaker critiques MIT’s high‑bar expectations and Wharton’s pragmatic analysis of AI ROI, urges the audience to ignore sensationalist headlines, and stresses that successful AI adoption is steadier and grounded in practical implementation.
- Team-Level Context Fluency in AI - The speaker stresses that organizations gain multiplied business value by teaching teams to articulate local, team‑specific context to LLMs and combining this with strong problem‑solving skills, turning domain uncertainty into actionable AI performance.
- Reversing Skill Ownership in AI - The speaker explains that, unlike traditional settings where team managers held problem‑solving expertise, the AI era flips this dynamic—individuals must now assume ownership of challenges while teams collectively develop the technical skills to leverage large language models.
- Empowering Individuals in AI Organizations - The speaker argues that AI‑native companies must place ownership of AI use at the individual contributor level, reshaping training and enabling teams to share prompts and custom GPTs to commoditize expertise.
- Cultivating Taste in AI‑Driven Work - The speaker explains that in the AI era “taste” means the democratized ability to identify and prioritize the most valuable problems, solutions, and learning methods, allowing teams to allocate effort where it yields the greatest organizational profit.
Full Transcript
# MIT vs Wharton AI Success Metrics **Source:** [https://www.youtube.com/watch?v=X7PWBlxJV1Q](https://www.youtube.com/watch?v=X7PWBlxJV1Q) **Duration:** 00:20:15 ## Summary - Conflicting AI ROI studies (MIT’s 95 % failure rate vs. Wharton’s 75 % success rate) are creating widespread confusion for businesses. - MIT’s unusually strict success criteria require measurable bottom‑line financial impact within a short timeframe, inflating the failure rate. - Wharton relied on executive surveys that include broader metrics such as productivity, time savings, and throughput, yielding a higher reported success rate. - The disparity between the studies is essentially an “apples‑to‑oranges” comparison, not a direct contradiction. - While MIT’s high bar emphasizes the need for rigorous financial validation, a balanced approach that also considers operational benefits would give enterprises clearer, more actionable guidance. ## Sections - [00:00:00](https://www.youtube.com/watch?v=X7PWBlxJV1Q&t=0s) **Decoding Conflicting AI Success Rates** - The speaker critiques wildly differing MIT (95% failure) and Wharton (75% success) enterprise AI studies, explains how methodological filters create contradictory headlines, and aims to give businesses a clear, realistic understanding of AI project outcomes. - [00:03:11](https://www.youtube.com/watch?v=X7PWBlxJV1Q&t=191s) **Balancing AI ROI Perspectives** - The speaker critiques MIT’s high‑bar expectations and Wharton’s pragmatic analysis of AI ROI, urges the audience to ignore sensationalist headlines, and stresses that successful AI adoption is steadier and grounded in practical implementation. - [00:06:21](https://www.youtube.com/watch?v=X7PWBlxJV1Q&t=381s) **Team-Level Context Fluency in AI** - The speaker stresses that organizations gain multiplied business value by teaching teams to articulate local, team‑specific context to LLMs and combining this with strong problem‑solving skills, turning domain uncertainty into actionable AI performance. - [00:10:19](https://www.youtube.com/watch?v=X7PWBlxJV1Q&t=619s) **Reversing Skill Ownership in AI** - The speaker explains that, unlike traditional settings where team managers held problem‑solving expertise, the AI era flips this dynamic—individuals must now assume ownership of challenges while teams collectively develop the technical skills to leverage large language models. - [00:13:26](https://www.youtube.com/watch?v=X7PWBlxJV1Q&t=806s) **Empowering Individuals in AI Organizations** - The speaker argues that AI‑native companies must place ownership of AI use at the individual contributor level, reshaping training and enabling teams to share prompts and custom GPTs to commoditize expertise. - [00:17:14](https://www.youtube.com/watch?v=X7PWBlxJV1Q&t=1034s) **Cultivating Taste in AI‑Driven Work** - The speaker explains that in the AI era “taste” means the democratized ability to identify and prioritize the most valuable problems, solutions, and learning methods, allowing teams to allocate effort where it yields the greatest organizational profit. ## Full Transcript
You know, I don't blame people when they
are confused about AI because the
studies that are coming out are also
confused. This week, October 28th,
Wharton came out with a study on
generative AI return on investment and
implementation at very large companies.
If that sounds like a familiar subject,
it should because MIT studied the same
group of companies just a few months
ago. The kicker is this. MIT study had a
95% failure rate on AI projects and
Wharton came back with a 75% success
rate. Now, it does not take a lot of
mathematical skill to figure out that
these are not compatible numbers. You
cannot be both correct. And so I want to
spend time unpacking what is really
going on at the enterprise, how we put
these two numbers together and what is a
reasonable path forward that cuts
through frankly the headline nausea that
I get from all of this just top lines
that don't make sense and that keep
changing all the time. Business needs
consistency. Business needs clarity and
business needs to be able to actually
build in a way that makes sense. So, my
goal is to ground you by the end of this
so that you don't get spun and confused
when people are saying, "Is it 75%, is
it 95%." Here's what's really going on.
95% came out of the extremely tight
screen that MIT put on Project Success.
That is one of the ways they effectively
engineered a headline that would go
viral. And yes, I'm just going to say
it. I think they engineered the headline
because the screen is tighter than
almost any other internal software
measure I have ever seen. In this case,
what MIT was saying is every project is
by default a failure unless you can
measure a dollar and cents impact on the
bottom line, not the top line of the
business within just a few months. It
was like 6 or 12 months or whatever it
was. If you can't do that, then it's
useless. That is no other software that
I have seen. If you're buying software
is measured that way. You always measure
it on internal metrics that you think
will map to larger business value. And
that brings us to the Wharton study and
the 75% success because Wharton took
more of that approach. Wharton's
approach was to talk to executives and
let executives tell them how they're
measuring ROI. And what executives said
overwhelmingly is that they're using
other metrics. They're not just using
dollars and cents on the bottom line.
They're looking at productivity. They're
looking at time saved. They're looking
at throughput. And when you look at all
of those, execs feel like you get a very
clear measure of success. And that's
where that 75% number comes from. So
really, if you want to know first what
the heck is going on and why they're
different, it's apples and oranges. You
have a very hard profit measure from MIT
and you have a looser, more conventional
software ROI picture from Wharton.
Here's what both of them are not getting
right and where I have sympathy. I think
MIT is correct that we need to hold AI
to a pretty high bar. This is a
transformative technology. It's also an
expensive technology. It is on the verge
of being 10x or more more expensive per
employee than any software was before.
Yeah, we're going to have different ROI
measures. So, I think that MIT is
getting at something when they're
challenging leadership to think
differently about software purchase ROI.
But I think Wharton is doing a great job
actually analyzing the the reality on
the ground and the way execs by
definition and by convention and the way
they usually act really measure stuff.
So, my ask to you, if I were to like
take all of this away, my ask is that
you not pay too much attention to these
kinds of headlines, I get inbounds,
right? I get emails, I get messages
coming in. I get it. It is confusing
when the news media loves to report
contradictory information. But the
reality at organizations that are
succeeding with AI is a lot more steady.
And that's the piece I want to leave you
with from a grounding perspective. When
we build with AI systems and they
actually work, there are a few things
that do align really well with both of
these studies and I'll sort of explain
that, but they don't like the studies
don't get at them, right? They don't get
at how to positively build and that's
you know me, right? Like that's what I
love to do. The first piece that I want
to lay out for you, think of these as
sort of building blocks that you can
build institutional fluency with. So, I
talked about individual fluency a couple
of weeks ago. I want to talk about
institutional fluency today. I think
that is one of the missing pieces that
connects these two studies. And I think
that understanding how it works will
help you to not get swept and pushed
around when the next whatever study
comes out with whatever number. The
biggest piece of institutional fluency,
if you want to set up a sort of whole
companywide fluency on AI, your company
has to get good at understanding and
shaping context awareness for teams and
individuals. And I think teams are
really the atomic unit here. Individuals
come and go, but teams are steady. Teams
have a particular vertical they take
care of. Teams have a particular domain
ownership. and institutions that are
fluent in AI understand that the value
of the team is the context they inhabit
and specifically the context they're
able to articulate to AI systems. So
when we talk about context engineering
typically that's a job. I'm suggesting
that we think of it less as a job and
more as everybody's job. Context is
something that we all bring to the
table. Context is something that teams
need to deliberately maintain. What do I
mean by that? Right? Like if you
understand at a very deep level, this is
the way my domain actually works. This
is the way I actually drive value for
the business. These are the unique
processes and workflows that I can use.
These are the areas of uncertainty and
the areas I need to explore in my domain
to get better. And if you can articulate
that intentionally to an LLM as a team,
you are going to be in a position to
deliver multiplied value to the business
relative to individuals working on AI
alone or relative to the work that we've
done before 2022 sort of pregenerative
AI. Context is king here. Context helps
us to
feed an AI with what's needed to be
useful at a local level within the
business. If you can't figure out how to
help your team to articulate context to
the AI, you're going to have trouble
with everything else. And this is one of
those things where like if you look at
the Wharton study and the success, part
of what's going on here is that leaders
are saying that they are seeing
accountable acceleration amongst teams.
Like the way I read that is that leaders
are starting to see teams pick up and
use context in their disciplines to
drive value and the executive just kind
of gets to measure it, take credit for
it potentially and count it as a
success. So context is the first piece I
want to call out. Institutionally fluent
organizations in AI understand that
context is local that context operates
at the team level not the individual
level and they are deliberately
fostering team level context fluency.
The second piece that I want to share
with you that institutionally fluent
organizations have is problem solving
skills. And this sounds really obvious
because we've been talking about problem
solving skills as an element in sort of
employee training and upskilling for for
decades, right? Way before generative
AI. But socializing those problemolving
skills is something that managers,
directors and above are coming to me
privately and saying this is really
hard. This is not easy. And I think that
part of why we see the discrepancy with
the Wharton and the MIT measures is that
the MIT measure, the 95% fail rate
measure demands that a entire
organization be so good at problem
solving that it meaningfully upshift the
bottom line. That is an extremely high
bar. You can get a whole bunch of teams
who are good at problem solving and if
you have two or three bad apples, you
will bottleneck somewhere in your
process and have trouble delivering
value to the bottom line. And so what we
need is we need to treat problem-solving
skills as a critical patch on team
fluency that we cannot live without that
we must have on every team and that we
will hire for if needed to get done. In
other words, AI problem solving is
becoming all of our problems today now
and it doesn't get better until we
actually fix it. So what does AI problem
solving look like in practice? We can
say it, but what makes it something that
a team can reasonably learn? Because
keep in mind, if teams know context,
teams are going to know problems and
teams are going to be able to sort of
learn to solve problems. I want to
suggest that problem solving is really,
if you peel the onion back and you think
about it deeply, a function of
understanding how AI thinks about and
processes information. Because if you
think about problem solving
conventionally before AI, we're really
manipulating information in order to
unlock ambiguous problem spaces. And so
traditionally, it would be like, I'm
going to write my product requirements
document or I'm going to do this data
analysis. And we're manipulating
information in order to get closer to
unlocking a complicated customer
experience or a painoint in operations.
And all of the stuff we talk about like
critical thinking skills, good writing
skills, those were all ways that we
could scale up manually so that we could
successfully manipulate information as
an individual and as a team to solve
these problems. And in that world,
individual skills mattered a lot because
individuals pushed information fluency
forward. Right? If the individual could
write well, they might write well enough
that the whole team was elevated, right?
And then ownership resided at the team
level. And so a team manager would be
responsible for would own solving the
problem, driving around obstacles, all
of that stuff you want good managers to
do. That is starting to flip. And I have
never shared this before. I think this
is really interesting. I think that what
we are starting to see in the age of AI
problem solving is instead the
individual needs to index really highly
on ownership and the manager or the team
needs to index highly on skills and
that's sort of a reverse of the usual.
So the problem solving skills, the
ability to understand how LLM works,
those actually can reside at the level
of the team, but the ownership piece has
to rest with the individual if we're
going to make progress. And I'll explain
why that flip has happened. When you
think about solving a problem in the age
of AI, what you really are doing is you
are understanding enough about AI to
feed the AI the problem in a way that it
could understand and work with. And I've
talked about this part before where
you're sort of chopping up the problem,
decomposing it so that the robot AI can
pick it up and manipulate the problem
and help you get through the problem
space faster, which is the whole goal.
It is easier to solve problems if the
robot intelligence is working on that
problem with us. Here's what I haven't
talked about before in practice with
real teams building real AI systems.
What I'm seeing is that ownership is
irreplaceable at the level of the
individual working with AI. If you don't
have a very strong sense as an
individual, as an individual contributor
of ownership and quality and assessing
the bar that AI is using to solve and
insisting that the AI isn't doing good
enough when it really isn't, you're not
going to be able to add any value at
all. Whereas in the past, you could have
that bar set at the team level and the
manager would be able to sort of manage
the informationational standard and it
would be okay because all of the humans
were working together and information
was moving slowly enough and we were
exploring the problem slowly enough that
the manager could act as a quality bar.
In this day and age, that's not true. AI
is giving everyone so much superpower
that you have to devolve ownership down
to the level of the individual
contributor. And I think that at root is
one of the reasons why organizations are
struggling so much with the AI
transformation. It demands more of our
individual contributors than it ever has
before. And we're not used to a world
where the individual contributor is the
atomic unit of the corporation as
opposed to the manager. Corporations are
founded on management theory. The idea
is that the manager is accountable to
for the domain for the department. They
are the representative of the business.
They work with the individual
contributor. That's how how we've done
it for hundreds of years. I am beginning
to think that that is not how AI native
organizations are actually going to be
configured. The power you have with AI
resides so heavily with the individual.
I don't think you can do it any other
way. I think you have to put ownership
at the level of the individual
contributor. And that has profound
implications for how we train people.
Because really what we need to train
people to do is you need to start by
taking ownership of your domain and your
situation, of your problems, of the way
you work with AI, of the bar you use it,
everything flows from that. And
ironically, what we previously had at
the individual level, this sort of
skill, hey, this is a really skilled
writer, right? This amazing writer, uh,
and we couldn't do it without him. And
he lifts up the whole team. That kind of
thing can now reside at the team level.
Look at how teams are sharing prompts
with one another. Sharing clawed skills
with one another. How teams are sharing
custom GPTs with one another. AI is
enabling the commoditization
of a lot of those skills. And when it
comes to AI problem solving, you can
encode a lot of the technical skills and
understanding of AI in sharable format.
And so let's say someone isn't super
familiar with how transformer
architectures work and how you want to
chunk problems so that the AI can read
the problem coherently. That's okay. You
write a prompt for them. You share the
prompt with the team. You have a brown
bag where you talk about what it does,
but they can just immediately run the
prompt and the skill translates and they
can gain skill over time as they
socialize with the rest of the team. But
what you can't do is give them the skill
and they don't have the sense of
ownership. That breaks. That does not
work. So we've talked about context.
We've talked about problem solving and
how it sort of inverts traditional team
and managerial norms. There's one more
piece that I want to talk about today
that I think underlies this concept of
institutional fluency that that isn't
talked about very often. I think that
previously the concept of taste, the
concept of is this excellent, is this
extraordinary, is this something that is
an incredible offer for the customer, we
could delegate that to a small, call it
a priesthood within the company. The
Steve Jobs of our company is over here.
He has taste. He's an extraordinary
builder. He's an amazing inventor. We'll
run this by him and that will be fine. I
think in the age of AI, taste is
something that doesn't work that way
anymore if you really want to move
quickly. And so, one of the things that
you want to do is actually give and
socialize a sense of taste down to the
team level so that teams are empowered
to move autonomously without sacrificing
extraordinary quality. And I think that
that quality tradeoff is one of the
pieces I really have been sitting with
in the Wharton and MIT studies. I feel
like MIT essentially had an extremely
high quality bar and Wharton had a more
relaxed traditional software quality
bar. And if you want to thrive and build
an AI native company that actually
works, you have to figure out how you
can socialize that insane almost founder
level obsession with quality and taste
to the point where the team has it built
into their DNA because they have so much
power with AI agents with uh AI tooling
to launch their own products to drive
their own corner of the business. This
might look different at different
companies. Maybe you say it's at the
department level, not the team level.
But the point stands, right? Taste is
something that shows up at a much more
democratized level than it did in the
pre-AI age. And what's interesting is
it's not just is this product good
taste. It's taste in problems. Which
problems are spicy that we should choose
to solve? It's taste in problemsolving
skills. Taste in learning methods. What
I'm saying is you have to develop a
sense of where the juice is in the
profitability matrix of the
organization. Maybe the most effective
thing your team can do is to scale up
for the next 3 months and other teams
don't need that but yours does. Maybe
the most effective thing you can do is
double down on problem space discovery
and other teams are building product or
maybe it's a more traditional definition
of taste and you're working on an
excellent product. But the reason that
matters is because the team has to have
taste or the tooling they're using with
problem solving with high um high
ownership is wasted. Taste is
effectively a fancy way of saying pick
the right thing to work on and make sure
that you are really really good at
knowing what good looks like. That's
taste. When we talk about someone with
high taste in fashion, they pick the
right thing to wear and they know how to
wear it so it looks good. very very
similar idea and I think that that's
something that we could previously
delegate to just a handful of people to
a tiny sort of collection of folks when
IBM was at its height IBM had taste
makers they were a group of 10 or 15
people who were licensed to break all
the norms of the organization and they
were licensed to do that by the
organization so that they could
introduce creative thinking well their
their taste has to be democratized that
idea does not work anymore more. We need
to build institutions that socialize a
sense of taste. And I do want to suggest
this is not universal. Right? The way
LLMs work is universal. The ability to
learn to solve problems with LLMs is
also a universal skill. The sense of
ownership is a universal skill. Taste is
not. Taste is specific to your vertical.
Taste is specific to your situation.
Taste is more like context, which I
mentioned at the beginning of this
video. taste requires you to know your
local domain very very well and have an
excellent taste in problems. So there
you go. I think what we're really
talking about between Wharton and MIT is
institutional fluency. And I think the
three keys are context and then the
ability of teams to start to flip the
traditional relationship between
ownership and skills. Ownership residing
now at the individual level, skills at
the team level and then finally taste. I
think that taste is something that we
have to push down into our organizations
and that's also new. What do you think
you're missing or I'm missing on AI
fluency in institutions? This is an
evolving field. I'm learning and seeing
this in real time. What are you saying?