Macro Trends Driving Data Lakehouse Adoption
Key Points
- Three macro‑trends are driving analytics modernization: exploding data volumes and costs, evolving data consumption patterns (especially AI‑driven use cases), and a disruptive shift in data architecture.
- Enterprises are spending significantly more—estimated ~30% YoY—not only on storing data across lakes, warehouses, and other stores but also on managing, governing, and securing the data lifecycle.
- Business users and AI applications are demanding faster, broader access to data for tasks ranging from automated advertising optimization to human‑in‑the‑loop credit underwriting, intensifying the need for timely insights.
- Growing regulatory and privacy requirements mean modern data platforms must embed robust security and governance to safely enable data sharing, while also delivering rapid “democratized” access for users.
Sections
- Untitled Section
- Data Governance and Architecture Evolution - The speaker highlights how rising regulatory demands and the push for rapid, democratized data access drive enterprises to adopt secure, governed data practices and flexible, cost‑optimized cloud storage architectures.
- Lakehouse as Open, Cost‑Saving Highway - The speaker describes how an open data lakehouse eliminates vendor lock‑in, cuts warehousing costs by up to half, and uses a highway/toll‑road analogy to illustrate flexible, low‑cost analytics.
Full Transcript
# Macro Trends Driving Data Lakehouse Adoption **Source:** [https://www.youtube.com/watch?v=N8TFwV4Q_w0](https://www.youtube.com/watch?v=N8TFwV4Q_w0) **Duration:** 00:07:58 ## Summary - Three macro‑trends are driving analytics modernization: exploding data volumes and costs, evolving data consumption patterns (especially AI‑driven use cases), and a disruptive shift in data architecture. - Enterprises are spending significantly more—estimated ~30% YoY—not only on storing data across lakes, warehouses, and other stores but also on managing, governing, and securing the data lifecycle. - Business users and AI applications are demanding faster, broader access to data for tasks ranging from automated advertising optimization to human‑in‑the‑loop credit underwriting, intensifying the need for timely insights. - Growing regulatory and privacy requirements mean modern data platforms must embed robust security and governance to safely enable data sharing, while also delivering rapid “democratized” access for users. ## Sections - [00:00:00](https://www.youtube.com/watch?v=N8TFwV4Q_w0&t=0s) **Untitled Section** - - [00:03:07](https://www.youtube.com/watch?v=N8TFwV4Q_w0&t=187s) **Data Governance and Architecture Evolution** - The speaker highlights how rising regulatory demands and the push for rapid, democratized data access drive enterprises to adopt secure, governed data practices and flexible, cost‑optimized cloud storage architectures. - [00:06:26](https://www.youtube.com/watch?v=N8TFwV4Q_w0&t=386s) **Lakehouse as Open, Cost‑Saving Highway** - The speaker describes how an open data lakehouse eliminates vendor lock‑in, cuts warehousing costs by up to half, and uses a highway/toll‑road analogy to illustrate flexible, low‑cost analytics. ## Full Transcript
In a previous video, we talked about the
data lakehouse concept and shared a
story about how data lakeous are much
like the operations of a commercial
kitchen in a restaurant. So, definitely
check out that video if you haven't seen
it. Today, I'd like to discuss more
about the key drivers for and values
delivered by an open data lakehouse
architecture as well as share a couple
examples. And to help me do that, I'm
very excited to invite Edward Calvas,
director of product management for IBM
databases to join us. Edward, thanks for
being here.
Hey, love.
So Edward, let's start with the major
macro trends that we're seeing and that
are leading organizations to modernize
their analytics infrastructures. How has
the use of data shifted drastically in
the past couple years?
Well, of there are three major macro
trends that we're seeing in the market.
First, the amount and cost of data is
exploding.
Second, data consumption patterns are
expanding and changing.
And third, data architecture is being
disrupted and transformed.
So, could you speak briefly about each
of these?
Sure thing.
So there's no doubt that the amount
of data is expanding rapidly, but also
it's coming from a variety of different
sources and in all sorts of new data
formats.
What this means is that to manage all of
this data, enterprises are spending more
money and some estimate that that is in
the range of 30% year-over-year.
Okay. So, when you talk about the cost
of data, are you just referring to the
cost of storing it in different
repositories like data lakes,
warehouses, or other stores or are you
also referring to the cost of managing
and governing the life cycle of that
data?
Well, it's actually both. Let's talk
about the patterns of data consumption.
There's an everinccreasing demand for
the use of data, especially from
business users. There's no doubt that
analytics
has become an essential component of
almost every job and certainly AI, the
use of AI is expanding rapidly.
Now, this doesn't mean that every
business user needs to become an AI
expert, but it does mean that more and
more we're seeing AI being used to
automate and optimize certain decisions
at scale, such as advertising campaigns
or supply chain networks.
AI is also being used to augment human
in the loop decision-m such as credit
risk underwriting.
This means enterprises are always
looking for more data
and use it to drive new insights,
right? And and you know what about um
the data privacy and data regulatory
concerns that are around AI?
Sure. When you combine this with
increasing regulatory standards,
enterprises will require higher levels
of built-in data security and governance
in order to enable this data sharing and
consumption. Absolutely. And you know
another thing that we hear a lot is the
democratization of data. So it's about
create it's about time to value right.
uh business users, yes, they need data,
but they need it like yesterday, right?
It doesn't do the user or the
organization a lot of benefit. If it
takes long complicated processes for
users to get access to that data so to
get the most value out of the data, it's
got to be consumed as quickly as
possible. Um and of all while still
adhering to those governance and
compliance policies. Would you agree
with that?
Absolutely. That leads us to our third
point, which is architecture.
Organizations are realizing that the way
the data is managed needs to change. The
emergence of commodity cloud object
storage
and the adoption of open data formats
is really allowing enterprises to
increase the return on investment uh in
data management. And they're doing this
through the optimization of the price
performance
of their workloads across different
storage and compute tiers. Now what does
that mean? It means that organizations
can benefit from having the right tool
for the right job at the right cost
instead of defaulting into a data
warehouse which may be appropriate in
some cases but can also become very
expensive and ineffective in others.
Right? So what I'm hearing is more data,
more users and more uses of that data,
right? And all while still uh better
ways to share and manage access around
it, right? Um so Edward, how are these
aspects related to the key values that
are delivered by an open data lakehouse
architecture?
Great question, love. We see three key
values that are delivered by an open
data lakehouse architecture. First, an
open data lakehouse provides the
foundation for users to easily and
cost-effectively access, store, manage,
and unify large amounts of data and from
different sources and in different
formats.
Second, an open data lakehouse can be
easy to deploy within existing
environments, providing users fast
access to more data without having long
procurement, onboarding, or data
pipelining and wrangling processes,
making it much easier to consume. And
third, open data lakeouses can optimize
your analytics workloads to run where
they perform the best and are most
costefficient. All as part of an
integrated architecture.
Wow. So, you know, I've noticed one
thing. You've said the word open a lot.
Can you explain to us what the
difference is there?
Sure. I'm glad you brought that up. A
lakehouse should leverage the
capabilities across existing data and
analytics environments. If you already
have data and analytics workloads in a
data warehouse or in a Hadoop data
lakeink, that's okay. You shouldn't be
forced to migrate or rip and replace
that environment in order to get started
with the lakehouse. But what about new
data and new workloads?
Well, a lakehouse should be the starting
point for new data and new workloads as
well as provide a modernization path for
existing environments over time.
You know, the other thing that we hear
uh often is customers getting stuck with
one vendor.
Well, al open also means that you're
always in control of your data and you
aren't forced into proprietary data
formats or specialized tooling in order
to use it. It also means that you can
maximize the use of your data without
having to make copies of it and move it
around. And this provides for lower
costs, higher productivity and better
governance, which ultimately leads to
what enterprises are looking for, which
is more trusted decisions.
Wow. So Edward, what I'm hearing is a
data lakehouse
is like a network of highways.
Some of them have tolls on them and some
of them don't like regular freeways. And
a lakehouse allows you to go on the toll
road, pay when you need to get somewhere
really fast, but when you're not in a
hurry and there's no traffic, take the
regular highway. Love, that's a great
analogy. Think about using a data
lakehouse to cut your data warehousing
cost by up to half by optimizing the
price performance of your analytic
workloads. or like you're saying, saving
money by driving on the freeway while
there's no traffic and you're not in a
hurry.
Wow, Edward, that sounds really
exciting. Let's hit the road.
Let's do it.
Thank you. If you like this video and
want to see more like it, please like
and subscribe. If you have questions,
please drop them in the comments below.