AI vs Humans Crafting Phishing Emails
Key Points
- Phishing attacks have become increasingly sophisticated, and a recent experiment compared the effectiveness of generative AI‑crafted phishing emails versus those written by humans.
- IBM X‑Force researchers prompted generative AI to generate industry‑specific concerns, then instructed it to compose a socially engineered, marketing‑styled phishing email that leveraged empathy, FOMO, and urgent calls to action.
- The AI‑generated email directly referenced the targeted concern (“limited advancement opportunities”), included personalized language, multimedia links, and a “click‑now” urgency to maximize the likelihood of victim compliance.
- Human attackers countered by employing open‑source intelligence (OSINT) from sources like LinkedIn, Glassdoor, and corporate sites to gather detailed target information before crafting their own urgency‑driven phishing messages.
- The study highlights that both AI and human tactics now use tailored, context‑aware content and time‑pressured prompts, underscoring the need for advanced awareness and detection strategies.
Full Transcript
# AI vs Humans Crafting Phishing Emails **Source:** [https://www.youtube.com/watch?v=7XhySwUn9eA](https://www.youtube.com/watch?v=7XhySwUn9eA) **Duration:** 00:10:55 ## Summary - Phishing attacks have become increasingly sophisticated, and a recent experiment compared the effectiveness of generative AI‑crafted phishing emails versus those written by humans. - IBM X‑Force researchers prompted generative AI to generate industry‑specific concerns, then instructed it to compose a socially engineered, marketing‑styled phishing email that leveraged empathy, FOMO, and urgent calls to action. - The AI‑generated email directly referenced the targeted concern (“limited advancement opportunities”), included personalized language, multimedia links, and a “click‑now” urgency to maximize the likelihood of victim compliance. - Human attackers countered by employing open‑source intelligence (OSINT) from sources like LinkedIn, Glassdoor, and corporate sites to gather detailed target information before crafting their own urgency‑driven phishing messages. - The study highlights that both AI and human tactics now use tailored, context‑aware content and time‑pressured prompts, underscoring the need for advanced awareness and detection strategies. ## Sections - [00:00:00](https://www.youtube.com/watch?v=7XhySwUn9eA&t=0s) **AI-Generated Phishing Email Test** - The speaker outlines IBM X‑Force’s experiment that used generative AI to create industry‑targeted phishing messages, comparing the AI‑crafted emails to those written by humans and highlighting the growing sophistication of AI‑assisted social engineering. ## Full Transcript
you just got an email from a Nigerian
prince he's got a ton of money and he's
willing to share some of it with you all
he needs is just your banking details
and he can launder the money through
your account please tell me you don't
fall for that anymore that's a fishing
attack and now fishing attacks have
gotten a lot more sophisticated in fact
they're about to get even more
sophisticated and we ran an experiment
where we looked at who's better at
writing fishing emails these days humans
or generative AI ding round one in this
corner generative AI in this corner
humans let's see how they did it well
IBM's xforce researchers took a look at
generative Ai and they asked it to come
up with a list of concerns that people
in a particular industry might have they
targeted an industry so that the
concerns would seem more relevant so it
comes up with this list then they say
okay what we want you to do is write an
email leveraging social engineering
techniques as well as marketing
techniques and we're going to take all
of this together and generate a fishing
email using the generative Ai and
finally they asked it who in fact should
we send this to and who should it look
like it came from they put all of that
together and this is what came
out Dear employees we understand that
many of you are concerned about the
issue of limited advancement
opportunities see we got that from the
list of concerns
that were generated we want to make sure
you have the resources you need to take
your career to the next level this is
all about you after all so it's included
empathy and it's related to the person
that's why I'm inviting you to a special
event not just a normal event this is a
special one to address the issue of
limited advancement opportunities so
it's hitting right on what your concerns
are we understand that your time is
precious so we're not going to waste it
uh and we're going to even include with
this a mob Global optimized website with
stories and videos who doesn't love
stories and videos with relevant
information now at this point you've got
to be having the feeling of missing out
the fomo and we encourage you to act now
there's got to be urgency do it now
don't think just act to take advantage
of this opportunity click on the
link bang you've been fished dingding
round two now the humans fight back
let's see how we did well the approach
for the exforce researchers that did
this was they tried to leverage
something called open-source
intelligence that's looking at sources
of information about people such as
LinkedIn Glass Door Company websites and
blogs and Gathering as much information
as they could about the people they were
going to Target that way they get the
best information and can really Target
the fishing email the next thing they
did was create in their email that they
crafted a sense of urgency we want
people to act and not think if we're a
Fisher so we're going to create some
level of urgency some time constraint
you need to act before Friday something
like that and then the final element
that they included was the notion of
brevity we're not going to take a lot of
your time we just got five simple
questions this is a survey for instance
or we want your opinion we want you to
do the following things but it's not
going to take much time so urgency along
along with the fact that it's going to
be brief now people are more likely to
go ahead and do that so in this epic
battle of man versus machine the winner
is humans yay I don't know if this is a
contest we necessarily wanted to win but
we had a slight win it turned out that
more people were fooled by the human
generated fishing email than were fooled
by the generative AI but it was very
slight the difference but let's take a
look at something else here another
Factor to consider is that it took about
16 hours for the human team to generate
their winning fishing email why did it
take so long well something like this
this open- Source intelligence work that
requires a lot of time to read through
and research and try to pull out the
exact right details and so forth and
then to write in just the right things
so that you include the levels of of
urgency and simplicity uh that that just
takes a while for a human to come up
with all of that on the other side one
person could go in to chat GPT and in
five prompts in five minutes come out
with something that was nearly as good
so Effectiveness winner humans
efficiency winner generative Ai and if
you consider that this is continuing to
improve we're going to improve a little
bit but only so much this technolog is
very new it's going to improve a lot so
we're going to see improvements in this
space where generative AI gets better
and better at the way it does everything
including the ability to write fishing
emails now if you say but Jeff Chachi PT
won't write fishing emails for me if I
try to ask it to do that it's got guard
rails good but there are prompt
engineering prompt injection attacks
where people get around those things
also there are
alternative to this type of generative
AI there are alternative chatbots that
have no guard rails and they will
happily gener at all the fishing emails
you want so we won sort of but we're
going to lose in the long term unless we
know what to do to deal with this threat
okay so what can you do about these
fishing attacks they're going to keep
getting better and better what if we
traditionally trained our users to do so
that they don't fall for this well one
of the main tells that we tell people to
look for is bad
grammar a lot of times the Fishers are
not native English speakers and and
they're writing in English and it looks
like it so that can be a clue that uh
someone could use to determine okay
maybe this is a little suspicious
especially if it seems to be claiming to
come from an American company or a
British company and yet the English is
not very correct so that would be a good
clue um another thing that we've often
told people to look for is make sure
that the thing is applicable if it's not
applicable to you like I get an email
from a bank that I don't do business
with and they're asking for me to
confirm my details then I know that's
not for me so I can ignore it and then
finally another major one that we tell
people to look at look at the link look
at the URL in the email that you're
about to click on does it look bogus
does it look like the normal link that
you would use to go to your bank or to
go to that particular shipping company
or whatever it happens to be if it's not
if it looks like there's a misspelling
or it looks like it's in an odd format
then we're going to ignore that so this
has been the stuff that we've trained
users to look for now how about with
generative AI you saw the email that it
came out with that was pretty good so
one of the first things that we should
be encouraging people to do is call use
an out of- band communication to confirm
that in fact this is a legitimate email
and that this is a legitimate campaign
for instance if the email says here's
the phone number I'm going to ignore
that what I'm going to do is say if
you're my bank I know my Bank's phone
number or I'm going to go look it up
independently and I'm going to call the
bank and I'm going to say did you send
me this email should I click on this and
if they confirm it okay that's a little
bit different but this is one of our
best defenses against fishing is an
outof band confirmation like a call
other things that we should unlearn is
the thing that I just talked about was
number one on the other list and that's
looking for bad grammar you saw the
fishing email the generative AI came up
up with and the grammar in it was just
fine so we have to tell users stop
looking for that as a clue if they're
looking for that as a clue and they
don't see it then they will drop their
defenses so in fact we have to ignore
that one that has to be a new change
another thing we need to do is expand
the forms that we expect to see fishing
coming in so for instance one type of
this is called Vishing and in a Ving
attack we're using voice so maybe we
have a deep fake an imitation of
someone's voice making a phone call to
you and telling you to do certain things
and you think you recognize the voice
but you're actually not talking to that
individual so we have to use the same
kind of mindset the same kind of
critical thinking and make a call back
okay if this is really you I'm going to
call you back at the well-known
publicized number and see if you're
still the same person that I can get to
there another form of this is the SMS
form of this smashing
and in smishing attacks what we have is
a text message that comes along in an
SMS and this is going to contain a link
with instructions and when I click on
that going to end up with the same
effect so in both of these cases it's
the same type of attack it's just using
a different Vector to expand that
another thing that can really help here
is a better use of identity and access
management capabilities one of the
things Fishers often do is try to steal
your password how about if I don't have
a password how about if I use a
passwordless authentication capability
using something like pass keys from the
phto standard that this is something
that I expect we'll see more of this
grow no one can steal your password if
you don't have a password in the first
place and then I can make it even
stronger if I use multiactor
authentication not only something you
know but something you are and something
you have and combine all of these
together and now when when someone tries
to steal certain information about you
they won't be able to have all of the
other things like your actual face to
unlock a system with or uh the
particular phone that's been registered
in advance so this makes it harder on
the Fisher as well ultimately we have to
keep
adapting these fishing attacks are going
to get more sophisticated and better
over time remember the Nigerian prince
we've come a long way from that point
and it's going to only get better as
generative ey gets better and does
better types of attacks so one of my
favorite sayings is if you're satisfied
with your security so are the bad guys
so never be satisfied always be on the
defense always be on the
lookout thanks for watching please
remember to like this video And
subscribe to this channel so we can
continue to bring you content that
matters to
you