Data Observability: Driving ROI Benefits
Key Points
- Data observability delivers ROI by helping both data producers (engineers, platform teams) and data consumers (ML engineers, analysts, scientists) detect and resolve hidden issues throughout the data pipeline.
- In a typical journey—ingestion → lakehouse transformation → warehouse storage → consumer access—subtle bugs (mis‑formatted records, transformation errors, duplicate loads) can silently corrupt data before it reaches analysts.
- Without observability, engineers spend 10‑30% of their time firefighting data quality problems, while consumers waste effort building models on unreliable data, leading to frustration and reduced productivity.
- Implementing an observability solution surface‑to‑surface these incidents in real time, allowing engineers to focus on building pipelines and consumers to trust the data for accurate analysis and ML training.
Full Transcript
# Data Observability: Driving ROI Benefits **Source:** [https://www.youtube.com/watch?v=j8X0xiHTW54](https://www.youtube.com/watch?v=j8X0xiHTW54) **Duration:** 00:12:06 ## Summary - Data observability delivers ROI by helping both data producers (engineers, platform teams) and data consumers (ML engineers, analysts, scientists) detect and resolve hidden issues throughout the data pipeline. - In a typical journey—ingestion → lakehouse transformation → warehouse storage → consumer access—subtle bugs (mis‑formatted records, transformation errors, duplicate loads) can silently corrupt data before it reaches analysts. - Without observability, engineers spend 10‑30% of their time firefighting data quality problems, while consumers waste effort building models on unreliable data, leading to frustration and reduced productivity. - Implementing an observability solution surface‑to‑surface these incidents in real time, allowing engineers to focus on building pipelines and consumers to trust the data for accurate analysis and ML training. ## Sections - [00:00:00](https://www.youtube.com/watch?v=j8X0xiHTW54&t=0s) **Data Observability ROI Benefits** - The speaker explains how implementing a data observability solution boosts ROI for both data producers (engineers/platform teams) and data consumers (analysts, scientists) by improving the end‑to‑end data journey from source ingestion through lakehouse to warehouse. ## Full Transcript
hi everybody today we are exploring the
specific benefits in Roi of a data
observability solution further building
upon a previous video which laid out
what data observability is and why it's
important in today's climate if you
haven't already watch that hit pause now
you can find the link in the description
below and if you have well then welcome
and let's get started I'm going to start
with a typical overview of data Journey
before we do that I want to call out two
personas within an organization that
we're going to be highlighting in our
example below those are data producers
and data consumers producers are your
data engineers and your data platform
teams whereas consumers are your ml
Engineers data analysts and data
scientists both of these groups are
going to see Mutual benefit as well as
unique benefit to each by implementing
an observability solution so like I said
let's get started with this overview of
a typical data Journey so as you can see
below we have our
sources um we have our Lakehouse we have
our warehouse and we have our access the
Journey Begins With the data engineer
ingesting raw data from various sources
into the lakeh house the data
engineer then transforms and loads the
data into the lake house performing the
necessary cleansing and
standardization that
data is then processed for storage
within the data warehouse the data
scientist can then access that data to
perform any
relevant models training analyses Etc
now this seems pretty straightforward
but what we're seeing um is actually
what we're not seeing which is the risks
and everything going on behind the
scenes throughout this data Journey so
at the data ingestion Point your data
engineer has ingested raw data from
various Source however unbeknownst to
them um a subtle issue arises during
that ingestion process where certain um
records are misconfigured or formatted
incorrectly as the data engineer then
transforms and loads the data into the
lake house unintentionally a
transformation script introduces a bug
that alters the value of a specific
column impacting Downstream
analyses the cleanse data that's then
stored in a warehouse however due to
misconfiguration certain data is
actually duplicated during that loading
process now the data scientist who was
so excited about conducting all these
analyses with the data that they've
received is unaware of all these
incidents that have occurred before and
is now completing models with inaccurate
and unreliable
data in this current state your data
producers are entirely overwhelmed and
they're constantly fighting fires and
your data consumers are also really
frustrated because they're unable to
perform the correct models that they
want to do neither of these personas can
focus on what it is that they're skilled
to do that's because their data is
unreliable so let's try now associate
some numbers with this common
scenario a typical engineer will spend
roughly 10 to 30% of their time just
uncovering data issues additionally
they'll spend again between 10 and 30%
of their time resolving those issues so
let's say 20% for both so based on a
40-hour work week we work about
1,920
hours
annually now if we multiply that by this
combined um 40% of
time in today's environment data
Engineers are spending approximately
777
hours just just identifying and
resolving data issues let's break that
down a little bit further let's say you
know on average a data engineer has a
annual salary of 100K that correlates to
about
$52 per
hour if we multiply the $52 by an hour
by the 777 hours that they're spending
that is
$40,000 that is being spent just
detecting and resolving data issues I
think we can all agree that is not a
very good use of
time data Engineers need to be able to
detect things earlier especially unknown
data incidents when they are reactive in
nature they're forced to rely on their
data analysts or data scientists to
uncover data issues this often means
that data qualities are discovered too
late or in in fact not at all data
observability is this more shiftleft
approach detecting things as they occur
at the source and allowing for you to
resolve them before it actually gets to
that access
layer the outcome of this and where we
see three cor improvements with a data
observability
solution are meantime to
detection
with a DAT of observability solution in
place meantime to detection becomes
almost
instantaneous most a most alerts that
fire are in real time the second one is
meantime to
resolution improving meantime to
resolution is all about helping the data
platform teams quick walk through the
context of the problem such as where the
problem is occurring why it's occurring
and then resolving it as quickly as
possible the last core metric is overall
enhanced data
quality all these things rule together
to improve things for your data consumer
data scientists and ml engineers rely
again on this high quality data to do
their high value tasks um like training
and deploying accurate models so by the
data producers using an observability
tool it helps them establish trust in
their data and really focus on those
high value tasks rather than wasting
time doing activities like uncovering
bad data and and and sending things back
to their
producers so we talked about earlier how
we got to this 40% number I'm just going
to jop them down here so that we recall
them so we said the average meantime to
detection between 10 and 30% we're
saying 20 again for same thing for
resolution and currently data quality is
low and
untrusted so now let's re-explore this
example of a data Journey but this time
imagining that you have a data
observability solution in
place so now now at your data ingestion
site at the source a data observability
solution immediately Flags any
improperly formatted records during that
ingestion data Engineers receive their
alerts in real time allowing them to
rectify the issues very
promptly once we get to the
transformation and loading section where
previously um we had a bug that altered
values in a specific column um these
again are instantly identified your data
Engineers have received a notification
allowing for them to correct um that bug
and ensuring that it's not impacting any
Downstream analyses with this in place
the data engineer resolves the
transformation in real
time in the data warehouse we had some
misconfigurations that caused data
duplication
these are now being detected in real
time um during that loading process and
again the alerts are notifying the
engineers allowing for them to make the
necessary configuration adjustments
preventing redundant data in the
warehouse lastly at our access layer our
data scientist is now working with more
transparent and reliable data set
therefore encountering fewer anomalies
during their model training and they're
now experiencing a smoother and more
efficient model training process with
improved data
quality so for each of these different
stages the alerts have been fired and
everything is quickly
resolved resulting in better data
quality at the
end if we bring this back to our little
calculation by implementing a data
observability solution your meantime to
detection can be reduced from
20% to 1% so essentially real time your
meantime to resolution will be increased
by 2x by surfacing the root cause
analysis and exposing any Downstream
impact so since this is 2x this will be
resolved to
10% and then lastly your data quality
that was previously very low and
untrusted now is
very
high and
trusted now if we come back to the
example below uh before um when we were
talking about how much time a single
engineer is spending just uncovering and
resolving data issues by implementing a
data observability solution and using
these um these new numbers that we have
around meantime to detection and
meantime to resol solution you will be
saving your data Engineers
600 80
hours of work
annually if we take that back to the
average salary of a data engineer the 52
hour $52 per hour that results in a
$333,000 call savings annually for
single data
engineer
so imagine the possibilities of what you
could save with a data observability
solution with an engineering team of 10
50 100 well lucky for you you don't
actually have to imagine it you can just
click the link in the description below
of this video and find out just how much
Roi a data observability solution can
bring your organization
today if you like this video and want to
see more like it please like And
subscribe if you have questions please
drop them in the comments below