AI's Limits: Novel Reasoning
Key Points
- The speaker stresses that AI, particularly large language models, are great at copying and re‑phrasing existing patterns but are fundamentally weak at genuine novel reasoning and solving brand‑new problems.
- LLMs don’t actually reason; they simply retrieve contextual information, and making them perform symbolic reasoning requires cumbersome tool‑chains, underscoring how hard it is to give them true reasoning ability.
- Their apparent intelligence is an illusion created by massive reading—LLMs sound smart because they’ve ingested vast text, yet extensive reading doesn’t equate to the capacity to tackle new, unseen challenges.
- Real‑world business judgment and many societal tasks rely on solving novel problems, a domain where humans still excel and AI remains a poor bet despite its impressive conversational fluency.
Full Transcript
# AI's Limits: Novel Reasoning **Source:** [https://www.youtube.com/watch?v=b9WehQ_5qeA](https://www.youtube.com/watch?v=b9WehQ_5qeA) **Duration:** 00:12:20 ## Summary - The speaker stresses that AI, particularly large language models, are great at copying and re‑phrasing existing patterns but are fundamentally weak at genuine novel reasoning and solving brand‑new problems. - LLMs don’t actually reason; they simply retrieve contextual information, and making them perform symbolic reasoning requires cumbersome tool‑chains, underscoring how hard it is to give them true reasoning ability. - Their apparent intelligence is an illusion created by massive reading—LLMs sound smart because they’ve ingested vast text, yet extensive reading doesn’t equate to the capacity to tackle new, unseen challenges. - Real‑world business judgment and many societal tasks rely on solving novel problems, a domain where humans still excel and AI remains a poor bet despite its impressive conversational fluency. ## Sections - [00:00:00](https://www.youtube.com/watch?v=b9WehQ_5qeA&t=0s) **AI's Struggle with Novel Reasoning** - The speaker argues that large language models merely recall and repeat learned patterns rather than genuinely reason, and overcoming this fundamental limitation is far from easy. ## Full Transcript
are you so tired of seeing so many links
and so much content around what AI is
good at or Panic inducing posts about
how AI is going to take jobs great
because this is not that this is a post
about how AI is actually really really
bad at certain things and it's not
something that's easily fixable it's not
that people aren't trying it's that it's
not super easy to fix and it's something
that if it were me I would be betting on
from a skills perspective number one is
the skill of novel reasoning so large
language models actually don't reason
they just recall context now you can
stick a tool chain on there and and you
can make them symbolically reason to
some extent if you do that off of the
large language model and then just Port
the results back in and make it talk in
English right so the fact that I had to
use all those words to describe that
should tell you people are having to
bend over backwards to make llms reason
at all and the reason why is that llms
are actually just good at conversational
flow they're good repeating text and
patterns that we understand because
they've read all of our text they've
read everything they've read everything
we wrote before the internet they've
read everything after the internet and
now they're reading stuff that they
themselves created so they're really
good at repeating text and that's
amazing and it allows them to do a lot
of things but mostly it allows them to
do things where there are repeatable
patterns that they can copy from and
then regurgitate in ways that are
technically new but not new kinds of
things not new kinds of problems and so
much of business is about solving net
new problems frankly it's not just
business government is about that
education is about that you're solving
net new problems we're teaching people
to solve net new problems we're solving
net new problems for society whatever it
is that is what humans are actually
really good at and llms are really
really bad at it they just don't do it
and the reason we get fooled into
thinking that they do is because
they have read so much they sound so
smart they've read so much they've read
more than I have that anybody else has
my library is Tiny
comparatively and we think that if
someone's well read they must really
understand how to do novel reasoning and
that's been a really reliable assessment
of human intelligence for as long as
we've been able to read and write as a
species and that's no longer true and I
think that's really kind of confusing
our brains because we have this thing
that is super well read that sounds
super smart when we talk to
it and it's still not doing novel
reasoning it's still not solving net new
problems
reliably just reading everything doesn't
allow you to solve a net new
problem and this is where what we call
business judgment comes into play that's
a really soft wishy-washy word but a lot
of it hinges around reliably solving net
new problems in ways that make sense in
the market and and regardless of whether
the role is a sea Suite role or an
individual contributor role every role
has some business judgment to it and
generally speaking the business judgment
parts are the parts that matter the most
and those are the parts large language
models are not good
at and so that should be encouraging to
you we're not going to run out of jobs
that require business judgment because
we're not going to run out of problems
that require Innovation to
solve all right the second skill that
llms are terrible at is realtime context
in fact the founders of businesses that
are building llms like open AI have
admitted that there isn't really a great
answer yet to how large language models
are supposed to handle real-time
breaking news this happened just in 2024
as we had breaking news event after
breaking news event and sech that were
supposedly built around AI or had AI
components including
Google did not reliably update their llm
answers when breaking news happened
because they're just not designed for
net new real world context they're
designed to read a gigantic context
window and synthesize
information that's a terrible way to
handle a net new fact it's just
bad and it extends Beyond breaking news
there's a fundamental problem with llms
in that they are really good at
synthesizing from a large quantity of
written text in the past and they are
really really bad at understanding the
real world realities of something
happening right now in your local
context for instance no llm is going to
be able to realize that it is raining
outside right now and therefore I do not
need to water my tomatoes
now you can use much simpler apps for
that there are apps that will measure
soil moisture and then choose not to
water the garden but that's not
Ai and the thing is those simple apps as
you already know because we have had
those for a long time do not really
replace jobs either there is something
about local context that is
irreplaceable as far as human brains are
concerned you need someone to sit there
and I'm going to go from like watering
the tomatoes to something that like a
human can do if you were sitting there
and you're trying to digest a bunch of
different slack messages and you're
trying to understand what your boss is
expecting you to do and you're trying to
understand what that jur tiet
says and you are making sense of it all
in your head that process that's hard to
describe llms are really bad at
it and also the simple apps that water
the tomatoes are terrible it because
they can't even understand it and so you
know we've had those simple applications
of technology for the last you know few
decades and we've had the more fancy
applications that are like language
related for the last year or two and
that's what we're all excited about and
scared about this large language model
effect and so you might think wow llms
can really understand the slack messages
and they can understand the J messages
and so maybe they can help me think
through and make sense of this and the
truth is
they can synthesize from it they can
make patterns from it but if you're
actually trying to solve a problem with
deep understanding of real world world
context they're not really very good at
it so for instance I have tried this and
llms do not reliably understand for
instance how to assess the interrelated
consequences of a Jura ticket that is
getting worked on slow the Team Dynamics
behind it the slack messages that are
coming in
periodically and all the other tickets
that aren't getting done and part of why
and that's a real world example right
like we've all seen that if we've worked
in software and part of why that is a
problem is because so much of that
context is hidden from a text
perspective most of the world we work in
actually doesn't just work on text
there's a lot of human context unspoken
things things between the lines llm are
not good at that because they can't read
it and so even if we got something that
was good at breaking news or good at
recent updates we still wouldn't have
something that's good at reading between
the lines because there's no text there
you have to read between the lines it's
real world context and llms are just not
going to be good at it but humans are
and that's something you can bet on too
okay third one AI is under opinionated
that's the third skill
set and what I mean by that is that AI
is designed for conversational flow it's
designed to have a conversation that
keeps going and that means it's actually
designed to mirror to you they've done
studies on this and llms tend to mirror
the opinion they think will keep you
chatting and F for once I actually do
not think that that is a social
algorithm designed to keep you addicted
to chat it may become that but I think
it's actually a situation where the
large language model is trained to
replicate patterns in your utterance in
ways that make sense based on its very
large training data
set and so it's going to come back with
something it think match it it thinks
matches and so it's inherently a
mirroring technology which means it's
inherently really bad at decision-making
because it's just going to come back and
say what you say which means it's not
going to give you a separate perspective
now now you can brainstorm with it and
it can help you expand your
understanding of your own perspective
absolutely it can give you a loosely
held summarization of some alternate
views
sure but what it will not do is take a
strong position that is tightly held and
say this is what I really think should
happen here because llms have no idea
about what decisions are they're not
built for decisions they're built for
conversation and so whether you work in
business whether you work in government
I don't care you are still going to need
to make
decisions humans need to make decisions
even if you're just living your life and
you don't have a job humans need to make
decisions and humans are actually really
good at saying it's either A or B I'm
going to pick a and this is
why and llms just replicate human
conversation that talks about picking A
or B but they don't have an
understanding of what a is or what B is
and they certainly don't have an
understanding of the
choice and so without that they're not
going to actually be making those
decisions they're not going to be
recommending decisions in ways that are
deeply reflective and deeply rational
and that is why even if you are drafting
with large language models The
Innovation the deep thought that comes
with making good decisions is still
going to have to come from a person it
there's just no substitute for
it that's just the way it is and that's
a good thing it means that there are
skill sets that humans can rely on all
right so let's wrap this up what are the
three skill sets that AI is not good at
that humans are really reliably good at
that you can bet on number one novel
reasoning every role has some degree of
Novel
reasoning and or nearly every role and a
I is just not going to get there large
language models are really really bad at
novel reasoning because they don't
reason at all and let alone reason over
new context they don't even know what
new context
is number two realtime context and this
is related to the new context piece but
I want to talk specifically about the
fact that so much of real-time context
is around silences it's around things
that are between the lines it's around a
felt sense the thing we call intuition
llms don't have that realtime context is
going to remain pretty much impossible
for them to
together and number three AI is under
opinionated it just doesn't have real
opinions it talks with you as if it does
but it doesn't those three things are
actually all
crucial for job success and always have
been and they're not going anywhere and
so if you are tired of getting link
after link of like AI is taking my jobs
just come back to this post or share
this post with someone who is worried
about their role to remind them that
there are skills and I haven't even
listed them all these are just three to
get started there are skills that are
going to keep mattering regardless so
don't give up hope