Turning Models into Production: Overcoming Deployment Hurdles
Sections
Full Transcript
# Turning Models into Production: Overcoming Deployment Hurdles **Source:** [https://www.youtube.com/watch?v=OejCJL2EC3k](https://www.youtube.com/watch?v=OejCJL2EC3k) **Duration:** 00:06:49 ## Sections - [00:00:00](https://www.youtube.com/watch?v=OejCJL2EC3k&t=0s) **Untitled Section** - - [00:04:48](https://www.youtube.com/watch?v=OejCJL2EC3k&t=288s) **Automated MLOps CI/CD Pipeline** - The speaker explains how CI/CD can separate training (GPU‑heavy) and deployment (container‑based) environments, add monitoring, and automatically retrigger model training when performance thresholds are breached, eliminating manual effort. ## Full Transcript
Have you ever been training a model only
to find that it never reaches
production? Well, 68 to 80% of models
that are trained and developed never
actually make it to production. Well,
I'd like to introduce something that
that will make training a whole lot
easier for you and your team and make
deployments uh much easier and will have
much less stress. And to illustrate
that, I actually have a story to give
some context. So my team was uh on a
crunch time with a project with getting
a model out. And so we finally got the
department to approve a GPU server for
us to be able to use some of these
larger language models like BERT and
Robera and things like that. So we got
on there, we started doing our work, we
went as fast as we could. We put our
notebooks on there. We're getting some
good results. We started getting some
accur some some pretty good accuracy.
And we continue developing until one day
we try to SSH into the server and we
can't. So only to find out that our mo
our department only paid for one month
on the GPU server. So what happened to
all the notebooks, all the uh data, all
the features that we prepared on the
server? Completely gone. And so maybe a
moment of silence for all the notebooks
that I lost in that. But um that pretty
much reflects what a lot of manual
processes what a lot of manual training
processes uh I mean uh look like right
now. So first of all you're usually
starting out with EDA and EDA is just
exploratory data analysis. Can we get
the data that we need to make this model
a success? So you're looking at you know
uh getting it from SQL databases or from
different teams that can give you an
export but somehow you're gathering all
this data. Once you get the data you
know that the data is not ready. the
data has to be prepped and so you're
looking at some time with data prep and
uh working in some of the gaps and
seeing if anything needs to be cleaned.
And from there you might move to feature
engineering might which might still be
uh within the same process uh as as the
same time you're doing your data prep
and you're creating and you're
transforming some of these um columns
and you're turning them into new
features that will help your model.
Well, once you have the features, you're
ready to train. And so training is
usually the next step. Training is its
own uh process. It's its own task
because you have to look into different
models. You have to look and see which
one is going to give you the best
accuracy. Which one is most applicable
to your problem. Is it NLP? Is it a
regression? Um that type of thing. And
the training starts. Once you get some
good models, you also have to do the
hyperparameter optimization depending on
the model. From there, you're ready for
deployment.
And deployment can be its own can of
worms because it's either using some
sort of API or it's got to integrate
with a front end or backend. And if
you're a small team like my own, you
might be the people writing both the
front end and the training. Um, so
you're doing all of that and then
finally you're ready for monitoring and
looking at how this is performing.
It's up on the deployment server,
however you uh decided to do your
endpoint and you just got to look and
see if this accuracy is good enough for
the business. But what'll happen is you
know entropy eventually your model is
not going to be as accurate as it needs
to be and so this process this whole
process starts again or your team is
tasked with a new model. But all of that
is manual and uh really adds a lot of
headache only for only 60 to 80% only
20% is 40 to 20% of them actually making
it to production. That's a lot of work
and I'm here to show you a different and
better way. So MLOps as you can tell
from the name implements DevOps
principles, DevOps tools and DevOps
practices into the machine learning
workflow. And so the beginning of DevOps
and the beginning of really any uh
development project is you're going to
start out with uh de the the the dev and
the EDA work. All of that at the end of
the day is code, right? You're writing a
notebook, you're writing some sort of
Python script or some RS script or Julia
or something like that. Uh but all of
that is code and you can put all of that
code in a source code repository. And
what that does is it opens us up for the
automation that's going to come next. So
we can actually go in two directions
from our dev and EDA you can first of
all um the deployment
if that's an API that you're writing or
a front end or something like that the
deployment can have CI and CD tools
applied to those commits that you're
putting into your repository on the on
the other side your training
can also take benefit from CI and CD and
I'm repeating those terms terms of what
what that means is continuous
integration and continuous deployment.
Uh which just means that every time you
make a commit on your uh repositories
automatically you can build and
automatically you can you can deploy
your deployment or automatically you can
push a model to start being trained.
Usually if you have uh the resources
you're going to want to separate your
training infrastructure from your
deployment in infrastructure and that's
because they're doing different tasks.
training, you're usually going to want a
GPU, some sort of highly parallel um
computation. And on deployment, you
might be fine with spinning up Docker
containers or something uh little little
containers that might have a load
balancer to just uh handle demand. But
from there, both of these can benefit
from uh monitoring.
So in DevOps, there's naturally a
monitoring tools just to make sure your
deployments are still live, just to make
sure your rollouts are happening. And
you can also see how AB tests are doing
and things like that. We can apply the
same ideas to your model, how is your
model accuracy, are there things that or
maybe trigger, let's say you reach 80%
accuracy, which is which is too low.
What you can do from that is
automatically trigger a new training
process. uh and it'll take the code,
it'll begin the new training on new data
and using CI/CD and using automation,
you can get a new model up on the
production server without too much
hassle. So just imagine how much stress
is gone from do this manually to going
to this automated MLOps type of
pipeline. I hope this helps and I hope
that you'll be able to see better
accuracies and much more speed whenever
you're training your models. Thank you.
Thanks so much. If you like this video
and you want to see more like it, please
like and subscribe. And my department
said if we reach 10,000 likes, they're
going to pay for another month on the
GPU server. If you have any questions,
please drop them in the comments below.