From Answers to Analysis: AI in Finance
Key Points
- An MIT study found that copying decisions from ChatGPT (or similar LLMs) significantly reduces the amount of mental effort people actually use.
- In finance and other high‑stakes fields, many users offload decision‑making to AI so they can claim credit for successes and blame the AI for failures.
- Most people ask LLMs simple “answer‑only” questions—something Google already excels at—rather than leveraging the models’ analytical capabilities.
- The current transition is from answer‑machine tools (like Google) to “thinking‑machine” LLMs, which should be used for high‑leverage, high‑value activities.
- Effective use of LLMs requires structured, analysis‑oriented prompts that provide all necessary inputs, allowing the model to synthesize insights rather than just deliver generic answers.
Sections
- AI Decision-Making Cognitive Offloading - The speaker cites an MIT study showing that copying AI-generated answers bypasses mental effort, and criticizes how people—especially in finance—use ChatGPT for simple “answer” queries to shift responsibility for outcomes onto the AI.
- Evaluating LLMs for Market Forecasting - The speaker outlines a real‑world experiment using strong prompts and statistical analysis of portfolio results to assess whether large language models can meaningfully analyze financial data streams, emphasizing that they are analytical tools—not financial advisers.
- LLMs as Rapid Scenario Simulators - The speaker explains how language models let users instantly model multiple financial, housing, and career scenarios—acting as cheap digital twins—to improve decision‑making, and advises using structured prompts rather than generic queries.
- Prompt to Reflect, Toast - The speaker encourages the audience to contemplate the discussed idea before concluding with a casual “cheers.”
Full Transcript
# From Answers to Analysis: AI in Finance **Source:** [https://www.youtube.com/watch?v=J95DmmvgjIE](https://www.youtube.com/watch?v=J95DmmvgjIE) **Duration:** 00:12:51 ## Summary - An MIT study found that copying decisions from ChatGPT (or similar LLMs) significantly reduces the amount of mental effort people actually use. - In finance and other high‑stakes fields, many users offload decision‑making to AI so they can claim credit for successes and blame the AI for failures. - Most people ask LLMs simple “answer‑only” questions—something Google already excels at—rather than leveraging the models’ analytical capabilities. - The current transition is from answer‑machine tools (like Google) to “thinking‑machine” LLMs, which should be used for high‑leverage, high‑value activities. - Effective use of LLMs requires structured, analysis‑oriented prompts that provide all necessary inputs, allowing the model to synthesize insights rather than just deliver generic answers. ## Sections - [00:00:00](https://www.youtube.com/watch?v=J95DmmvgjIE&t=0s) **AI Decision-Making Cognitive Offloading** - The speaker cites an MIT study showing that copying AI-generated answers bypasses mental effort, and criticizes how people—especially in finance—use ChatGPT for simple “answer” queries to shift responsibility for outcomes onto the AI. - [00:06:26](https://www.youtube.com/watch?v=J95DmmvgjIE&t=386s) **Evaluating LLMs for Market Forecasting** - The speaker outlines a real‑world experiment using strong prompts and statistical analysis of portfolio results to assess whether large language models can meaningfully analyze financial data streams, emphasizing that they are analytical tools—not financial advisers. - [00:09:42](https://www.youtube.com/watch?v=J95DmmvgjIE&t=582s) **LLMs as Rapid Scenario Simulators** - The speaker explains how language models let users instantly model multiple financial, housing, and career scenarios—acting as cheap digital twins—to improve decision‑making, and advises using structured prompts rather than generic queries. - [00:12:51](https://www.youtube.com/watch?v=J95DmmvgjIE&t=771s) **Prompt to Reflect, Toast** - The speaker encourages the audience to contemplate the discussed idea before concluding with a casual “cheers.” ## Full Transcript
Do you remember the study that came out
of MIT that talked about AI taking away
people's brain power? It was it made a
lot of waves. Basically, the TLDDR is
that when it you copy and paste
decisions out of Chad GPT or thinking or
writing out of Chad GPT, it turns out
not much of your brain gets used. Big
surprise. There is a larger lesson
learned here. In a lot of high-profile
cases, and I'm noticing especially in
finance, people want AI to take the
burden of the outcome off their
shoulders. They want to give it to AI
and ask AI to make the decision for them
so they can take credit for being smart
and using AI when it goes well, stock
goes up, or so they can blame the AI
when the stock goes down. I have watched
people mess around with chat GPT and
they're not asking strong prompts.
They're not asking analytical prompts.
We'll get into how you do this better.
They are just saying, "Give me answers.
Should I refy at 6.2%. Is now a good
time to refinance my mortgage? When do I
sell my house?" Or, "How much money do I
need to buy a house? If I want to move
to this city, how much money do I need?
I want to negotiate with my employer for
a raise. How much should I ask for?"
These are answer questions. They're what
I call domain completion questions.
Google is actually very good for this.
This is what Google was designed for.
Google was designed to work with your
brain's propensity to ask for answers.
It is the answer machine. And one of the
things that we are all living through
right now is a transition from answer
machine like Google to thinking machine.
Blat GPT claude Gemini etc. We need to
make sure that we prioritize moving to
thinking machines for high leverage,
high value activities. Finance sure
comes to mind. It seems pretty valuable
because when you ask for answers from an
LLM, the LLM is trained to be helpful
and gives you a perfectly acceptable
generic answer because your prompt did
not give it the room to do what it does
best. your prompt did not give it the
room to analyze to actually dig deep.
And so what I want to focus on in this
video are the principles. I'm going to
use finance as a lens because I think
it's a high leverage high value
activity. We all do it one way or
another. Whether we're in a budgeting
app or whether we're investing or
whether we're negotiating compensation,
we have to do with money. I want to talk
about how you use LLMs for this because
it's a lens into how we use them for
high-v value thinking activities. And
the frame I want to propose is that it
is most effective if instead of using
domain completion or give me the answer
type questions, we change and we think
of it as give me an analysis given all
of these inputs that I'm going to give
you. And we very carefully structure the
prompt so that you can actually have a
correct place for all the analysis that
the LLM will need to complete in order
to give you a reasonable overall picture
of the decision you're contemplating. So
the goal is that the LLM is a
synthesizer. The LLM is a conversational
partner where it can process inputs more
efficiently than you. It can look at a
discounted cash flow sheet maybe more
efficiently than you, unless you're
Warren Buffett. If Warren's listening
from beyond the grave, hello Warren. But
the point here is that most of us don't
use it that way. I want to suggest that
this is because we are very
uncomfortable with uncertainty and using
LLM this way extends the uncertainty
runway to a degree that is difficult for
most of us to handle. If you use the LLM
to analyze your refinancing position,
you don't get a decision back. All you
get back is a set of options with a lot
more color, a set of options with a lot
more clarity around the details. You
have given the LLM a lot of information,
maybe your W2s, your 1099 income,
whatever you have, right? Your current
rate on the house, what you want to do,
etc. And you're going to get a lot of
options back. And the LLM may have an
opinion, and that may be okay, and you
may or may not agree, but it doesn't
give you an answer because you didn't
prompt it to be an answer completion
machine. And so you have to sit with the
uncertainty and the responsibility of
the decision. This is the part where the
lens zooms back into the wider world. We
need to get a lot more comfortable with
uncertainty and LLMs. LLMs are wonderful
analysis tools. We need to take that
analysis, own the decision and own the
consequences of that decision. That is
going to give us much much better
results. It enables us to harness the
incredible power for processing tokens
that LLMs put at our fingertips and not
reduce all of that power down to just
one decision. You actually want all of
that power on exploring optionality. And
the prompts that I've developed for
finance as a way of exploring this, do
just that. And you can get them on the
Substack and you can run them yourself
and you can see they're designed to
unlock analysis for various specific
financial scenarios. But I didn't stop
there. I thought it really has more bite
to it if I actually run a live
experiment. And so I am running a live
experiment using a small amount of real
money on Robin Hood and on Kshi the
events market. In both cases, I'm asking
three separate LLMs to formulate
opinions, analyses, establish bets on
specific trades that they want to
execute. We will then run those trades
and we will tell the LLMs that we will
judge the results in 90 days. 90 days
feels very short from a investing
perspective, from a call sheet
perspective, but it's also something
that we can sort of get directional on
and giving them the horizon gives the
LLMs a chance to plan for short-term. We
will see how we do. I've selected uh 03
Pro for this, Opus 4 for this, and Grock
4. I'm going to write it all up. I'm
going to have a copy of what they
predict. We're going to track it. We're
going to see how they do. The point here
is not by the way pick the model that
makes the stock go up. I know cases
where companies are AI washing their
financing and basically saying in AI and
the stock will go up. Brr. It doesn't
work that way. Stocks are not money
printing machines in the hands of AI.
Instead, they can process a lot of text.
They can provide useful context. They
can provide useful analysis to humans
that make decisions. And so my question
is actually do we have given a strong
prompt good data from a real life test
that helps us to understand how LLMs
interact with data streams and make
recommendations against real life
markets with real life consequences and
we're going to find out and at the end
of the time like I fully expect that
some of these will not work some of them
will work. We are going to see if any of
them actually end up coming out ahead of
the ledger. And we are going to see
whether there are substantial
differences, meaningful differences
between these models. And I may well run
actual statistical analysis on the
ending balance differences to see if
they are within a normal distribution
range from each other or whether they
are actually substantially outside the
confidence interval. I used to do a
little bit of statistical analysis and
that would be quite fun for me. That's
the point, right? The point is not Nate
is going to then pick this the the LLM
that makes all the stocks go up and
everyone's going to be happy. LLMs are
not financial adviserss. I'm not a
financial adviser. I am here to help you
reframe how you think about LLMs and get
you into an analysis space. When you
look at the prompts, I want you to think
about the prompts as tools for analysis.
How do you take a tool that starts with
here are the relevant inputs here is
your role here is the silent reflection
I want you to do hidden chain of thought
so that you can understand what the task
is whether you have all the inputs etc
here is the output I want here is the
success criteria here is your fallback
or rejection criteria all of these
things put them together into a
structured prompt around a particular
decision around whether or not you
should sell your stock if you become one
of those employees that get a stock
event, right? Like it's a specific
decision event, you can craft a prompt
for it. Buy a house, you can craft a
prompt for it. Start a new job, you can
craft a prompt for it. And so part of my
goal here is to basically lay out enough
of these examples that you can start to
take them and make them your own. You
can start to take them and say, "Where
do I need more analysis?" And maybe it's
not finances. Maybe it's picking a
college. Maybe it's picking an MBA
program. I sometimes have people weigh
in and they're like, "Nate, tell me the
best AI program to take." And I'm like,
it's hard for me to tell. Like, this is
actually a great example where you
should use a well structured prompt.
It's a high-value decision. It should
not tell you what to do. It should give
you the tools to overcome decision
anxiety if you can sit with a discomfort
of working through an analysis. And so,
what we're really asking for is not
investment advice. It's really can you
keep responsibility for the outcome
inside you and ask the LLM to give you
strong analysis for whatever the
decision is. In particular, LLMs are
very strong at analyzing wide ranges of
textual input and they're very strong at
developing alternatives and working
through alternatives. So, you can get a
wider picture than you would get from
most humans because it will read more
and it will look at more options if you
frame your prompt correctly. And so one
of the things that I think is sometimes
helpful in these situations is you don't
just write one prompt like you write the
prompt and then you tweak some of the
inputs. What if this scenario changes?
What if that scenario changes? That is
literally one prompt away. And it used
to be nearly impossible to get. It used
to be that if you sat down with a
financial adviser, with a real estate
person, with a career guidance person,
you would spend the entire hour and
whatever money you were going to spend
working through just one scenario
scenario at a of granularity than you
can do with one chat in an LLM and now
you can have 10 chats. Do you want to
model a scenario where you put 25% down
on the house, 15% down on the house? Do
you want to model scenario where you
take the job as a marketing manager at X
salary or Y salary? and what that does
to your budget. You can do all of that
in a chat if you're willing to live with
LLMs as analyzers. They free you up.
This relates to what I've talked about
with LLMs as digital twins. It's the
same concept. You are using LLM to
cheaply model future timelines. And when
you can do that efficiently, it helps
you greatly improve the quality of
decision-making. And I've chosen
finances to illustrate it because people
find finances very tangible and it helps
make it more real. And so, I've got the
prompts out there. You can look at them
if you want. If you just want to take
this away though, I want to again
challenge you to use a structured
thinking framework in how you interact
with AI. Do not do domain completion
questions like Google. It is one of the
biggest super tips I have for people
with prompt questions. Don't prompt chat
GPT like you prompt Google. It will not
work well. You will not get optimal
results. Instead, use the power of the
AI by asking it to actually think. That
means you have to retain and put on your
thinking cap. You have to own the
outcome and you have to challenge it
with a structured thinking framework.
And that's what my prompts are designed
to do is basically lay out how you do
structured thinking frameworks so that
you can make the most of the power that
AI requires. Yes, this is harder. You
have to gather actual data. You have to
sit down with prompt craft, which I can
help with, but like everyone's prompt is
going to be slightly different. And the
analysis approach frontloads the
uncertainty. You have to deal with the
uncertainty up front. That's okay. You
can get into that pattern now. You can
get into it with finance. You can get
into it with whatever frame you can with
a big decision. And then it becomes
second nature and you will use AI more
and more that way. And that unlocks a
ton of downstream benefit for you. If
you are the person in your life using
LLMs like this, you are going to
progress faster because you can
literally see more future timelines. It
is like a hidden superpower that is one
chat away. The goal is not to avoid AI
for financial decisions. I don't want
you to hear this and say, "Well, Nate's
going to run a test on Grock 4 and Opus
4 and and on chat GPT, and we'll see,
and if it goes well, then we'll use it
for financial decisions, and if they
lose money, then we won't." The beauty
of this is that we learned something
either way. The point wasn't the
decisions. The point was the analysis
and the relationship between the prompt
and the results. The point is to use the
LLM as a thinking partner. So that's my
challenge for you. The choice is
ironically very much yours. Can you use
AI as a thinking partner rather than
just using it for domain completion?
Think about it. Cheers.