Seven Pillars of Storage Observability
Key Points
- A world‑class observability tool is essential for storage arrays, just as a dashboard is critical for safely operating a car.
- The tool must address seven “pillars” of observability: availability, performance, capacity, security, inventory, cost, and sustainability.
- Each pillar answers key operational questions such as whether the storage is up, meeting latency/IOPS goals, has enough space, is protected from ransomware, what assets exist, how much it costs, and its power‑and‑carbon impact.
- By providing answers to these pillars, the observability solution enables proactive planning, risk mitigation, and alignment with financial and environmental targets.
- Core use cases—starting with end‑to‑end visibility—demonstrate how the tool helps administrators manage and optimize storage infrastructure across the entire lifecycle.
Sections
- Untitled Section
- Sustainable Storage Observability Use Cases - The speaker outlines how a storage observability tool supports sustainability goals and operational efficiency through visibility, proactive issue detection, performance optimization, capacity planning, and cost management.
- AI-Driven Storage Observability - The speaker explains how AI helps storage admins by automatically detecting anomalies, forecasting capacity, performing root‑cause analysis, and automating actions to reduce noise and manual effort.
- AI-Driven Storage Observability Stages - The speaker explains how AI advances storage observability—from basic monitoring to self‑learning workload tiering and finally agentic AI Ops that automate actions for optimized performance and cost.
- AI‑Driven Self‑Healing Storage Ops - The speaker explains how a chatbot‑style agent can deliver instant, goal‑driven responses and automatically remediate storage problems, creating self‑healing infrastructure that eases administrators’ workload.
Full Transcript
# Seven Pillars of Storage Observability **Source:** [https://www.youtube.com/watch?v=QGrLG_zqxJU](https://www.youtube.com/watch?v=QGrLG_zqxJU) **Duration:** 00:18:03 ## Summary - A world‑class observability tool is essential for storage arrays, just as a dashboard is critical for safely operating a car. - The tool must address seven “pillars” of observability: availability, performance, capacity, security, inventory, cost, and sustainability. - Each pillar answers key operational questions such as whether the storage is up, meeting latency/IOPS goals, has enough space, is protected from ransomware, what assets exist, how much it costs, and its power‑and‑carbon impact. - By providing answers to these pillars, the observability solution enables proactive planning, risk mitigation, and alignment with financial and environmental targets. - Core use cases—starting with end‑to‑end visibility—demonstrate how the tool helps administrators manage and optimize storage infrastructure across the entire lifecycle. ## Sections - [00:00:00](https://www.youtube.com/watch?v=QGrLG_zqxJU&t=0s) **Untitled Section** - - [00:03:18](https://www.youtube.com/watch?v=QGrLG_zqxJU&t=198s) **Sustainable Storage Observability Use Cases** - The speaker outlines how a storage observability tool supports sustainability goals and operational efficiency through visibility, proactive issue detection, performance optimization, capacity planning, and cost management. - [00:07:12](https://www.youtube.com/watch?v=QGrLG_zqxJU&t=432s) **AI-Driven Storage Observability** - The speaker explains how AI helps storage admins by automatically detecting anomalies, forecasting capacity, performing root‑cause analysis, and automating actions to reduce noise and manual effort. - [00:10:30](https://www.youtube.com/watch?v=QGrLG_zqxJU&t=630s) **AI-Driven Storage Observability Stages** - The speaker explains how AI advances storage observability—from basic monitoring to self‑learning workload tiering and finally agentic AI Ops that automate actions for optimized performance and cost. - [00:13:39](https://www.youtube.com/watch?v=QGrLG_zqxJU&t=819s) **AI‑Driven Self‑Healing Storage Ops** - The speaker explains how a chatbot‑style agent can deliver instant, goal‑driven responses and automatically remediate storage problems, creating self‑healing infrastructure that eases administrators’ workload. ## Full Transcript
Imagine you are buying a car.
And your car does not have any dashboard. So, what will happen?
Similarly, when you buy a storage array, you need to have an, uh, a very world-class
observability tool because buying a storage without observability is like driving a car
without a dashboard. You may keep moving, but you do not know when you are going to run out
of fuel, when you-your car is going to overheat, or you are headed for a
breakdown. Now, having said that, now that uh, I have established that we
need a world-class observability tool when you buy a storage box. What are the critical
operational questions that this tool can answer for you? And that I call it the seven
pillars of observability. So what are those pillars? Number one:
availability. Is my uh,
storage infrastructure available? Does it have enough availability for my applications? This is
the very basic question a storage admin needs to know. Hence that is the first pillar.
Second: performance. Is my
storage uh, infrastructure performing well? Does it give enough resources to my applications? How is
it doing with respect to latency and IOPS? These are the critical questions that a world-class
observability tool can answer once the storage array is configured. Now I moved into a very
critical one. The third one: capacity. Do
I have enough capacity now? What is that I was capacity I was using in the past? When is
my uh, my box running going to run out of capacity? How can I order more capacity when it's going to run
out? Next, and the very important pillar in the
recent days has become security. Is my storage infrastructure secure? Do I h, do I have
enough safety against the ransomware attacks? Is my security posture configured correctly? How do I
know this tool can answer those questions and make me safe? The next one:
inventory. What does my inventory look like? How many block storage devices I have? How many file
storage devices I have? How many of them are hyperconverged? All these critical questions so that I
can plan my storage infrastructure better can be answered when I know what my inventory has,
right? Cost. What is the current cost I am paying for my
storage infrastructure? How much is my cost going to escalate? Is my cost going to go out of uh,
a limit? Only my world-class observability tool can help, and hence this is a very critical part.
And last but not the least, sustainability.
Is my storage infrastructure doing well with respect to power consumption? What is
it taking for carbon emissions? Is my infrastructure aligning with my carbon goals?
What does the data say? Now, this is where my storage uh, observability tool will
help align my box. Now, having answered all the critical operational
questions via seven pillars of observability, let's look at use cases where the observability tool is a
must-have for storage boxes. The use cases that can help the admin to manage your storage
infrastructure better. What are those use cases? End-to-end visibility. In a
storage infrastructure, there are many components, from the host to switches to the storage box. And you
want to know all the paths from your applications and till the storage, what's exactly the
thing. So it helps teams understand how storage systems, applications and infrastructure are
interconnected to eliminate b-blind spots. Next, proactive issue
detection. By monitoring health performance and anomalies in real time, this enables early
detection and faster resolution of if-issues before the impact. Right?
Performance optimization. For a big time a storage admin has a problem: How is my
latency? How is my IOPs? How is my throughput? Now, if you have world-class observability tool, it can
provide insights into the workloads, bottlenecks and utilization patterns, ensuring optimal
performance and resource allocation. Next, capacity planning and cost
management. How am I doing with respect to capacity? With a world-class storage observability
tool, the usage trends and predictive analytics, a storage admin can plan the storage
growth, avoid overprovisioning, which is critical, and reduce unnecessary costs.
Right? In a data center today, incommon, data centers, they're generally multi-vendor
and hybrid environments. A world-class storage observability tool simplifies management across
different storage vendors and if whether it is on prem or cloud,
ensuring consistency and reducing complexity. It improves
reliability by continuously monitoring, reducing the downtime risks,
and enhances resilience against failures because of multiple upgrades or multiple
security threads. With all this, what you have in your hand is data-driven
decision-making. A world-class ob ... storage observability tool translates raw storage metric
into actionable insights
so that it can empower IT leaders to align their storage strategy with
their upcoming goals. I will go ahead and talk about the role of
AI in data storage observability, right? AI plays a huge part when
it comes to ease of storage observability. Observability is about managing
the data the storage array produces. And the entire infrastructure produces exabytes of data.
It's manually impossible to observe the data with m ... with a manual
way. Hence, the storage admins leverage AI in order to make storage observability and
management better.How, how does it help? Let's, let's go to the use cases. The first one: anomaly
detection.
AI algorithms in observability tools can learn normal storage behavior and automatically
detect unusual patterns. Example, sudden latency spikes, abnormal I/O or capacity
anomalies before they become critical issues. Second:
predictive analytics,right?
AI can forecast storage growth, performance trends
and proactively do capacity planning, enabling
proactive management and risk mitigation. Third
is root cause analysis. A pain
point for all storage admins. Instead of manually sitting through logs and
metrics, storage admins can leverage artificial intelligence, which can correlate
signals and across the infrastructure layers to
quickly pinpoint to the root cause behind the performance issues
or availability issues. And thus, it can help make storage
admins' life a lot simpler. Intelligent automation.
We all want our storage boxes to be plug and play, and AI can help us achieve that. AI can recommend
or even trigger actions based on the
data, observability data, and do load balancing,
or cache optimization without any human intervention, thus
reducing downtime and any manual operational effort.
Now, another big pain point for storage admins is a lot of noise.
All the storage boxes produce a lot of alerts because of all the components inside it. And in
large environments, storage admins are flooded with alerts. Here, AI comes to your rescue.
AI can filter the false positives and
prioritize what is critical and surface
out only what truly matters, thus achieving noise reduction.
Workload optimization. Are my
workloads running in the most efficient manner in a storage box? Well, it's very
difficult to figure out in a manual way, but observability tools with AI can analyze
user usage patterns and can suggest optimal data placement across
the entire storage box to-to ensure that the tiering is in
there to balance cost and performance. With all this in place,
AI is also learning from the data that the storage boxes are producing,
enabling in what I call as self-learning insights. So over the time, AI
adapts to the unique capacity characteristics of your storage box
and it then uses those characteristics of your workloads and storage systems
in order to improve the observability outcomes. With, now that we have gone through the role of
AI in storage observability, let's see about how agentic AI Ops
can help in storage observability much better. Now, before going there, I would
like to talk about four stages of storage observability, right? Now, first is
monitoring, the basic monitoring of a storage box. Second
is when I start observing the data that the storage box
produces and giving insights to the admin.
Third one is AI Ops, that I leveraged AI in order to make the storage
observability experience a lot better for the admin. And then, finally,
agentic AI Ops, where I have asked few agents
to do the work for me, which with normal AI I was unable to do. Now
what are those use cases where agentic AI Ops can take the
observability experience a notch better, right? Now, the first one: autonomous
monitoring and response, right? Now, what does this
mean? Based on the data that the observability tool is getting from the storage, it
can start responding real time to the storage admin. I am getting a message
that my drive is not available. So what do I do? What is my action?
So it can, in a chatbot kind of manner, provide immediate response and thus help storage
admins' life a lot easier. Second, a goal-driven response,
or goal-driven
operations, I must say. An admin can define the goals that
are needed from the observability, and when the goals are reached, the ... the relevant
agent starts responding with the admin, hence achieving the goal-driven operations. For
example, if he needs a performance latency less than
10seconds. So every time it exceeds 10 seconds, it goes and autocorrects
and thus, helping the admin, not having to sit through and monitor through thousand alerts and
doing those actions, rather trusting the agent to go see the goal and take the necessary
actions. Right? Now, next and the most critical and which is currently
not appreciated by all the users, is self-healing infrastructure. Now,
a storage box is a complicated piece of hardware with lots of components in it,
and lots of software that interact with each other to give the best data storage experience. So
it is going to come up with issues regularly. When the issues come up, admins run around in a team to
figure out how do I resolve the issues? But, if
I can, if I am able to outsource it to agents and then they can
self-correct themselves, I, we're basically achieving self-healing infrastructure. So whenever,
let's say, for example, there is a storage drive. And that failed, right? What can I do? A
controller went for a toss. What can I do? A back plane had some issue. How can I
heal it? So, all the actions and then automated manner,if it, it goes
for a toss, how can I self-heal them so that I can provide a best storage
experience for my admin. And last, and the biggest pain point, which is still not
solvable by AI completely, where agentic AI can really be a lifesaver is uh, it's in lifecycle
management. Now, a storage admin who has 50 to 100 storage
arrays has to regularly upgrade patches and the new releases, the new security
fixes. And every storage vendor has a different lifecycle, different release
timelines. So, today, admins maintain everything in an Excel sheet,
manually, interact with their team: what-what time it is coming up, then they have a
downtime, they have to communicate. And it is a management nightmare. Rather, you can outsource
all of it to agents. The observability tool can release the agents based on
the certain storage boxes, and they can achieve a complete lifecycle management
experience, where the agents will automatically track the amount of arrays, their release
timelines, and depending on each of the release timelines, they will take the necessary upgrade
actions and send an email once that is
done. If the upgrade has not taken place, they can send a communication
on wh-what the next upgrade should be and how it can be achieved, providing
a complete automated way of asset lifecycle management, eliminating hours and hours of manual uh,
effort that the storage admin puts today. And here agentic AIOps can be really a lifesaver.