Scaling Kafka Event Endpoint Management
Key Points
- Use the AsyncAPI specification to formally describe Kafka event topics, giving them the same developer‑friendly interface and documentation standards as traditional APIs.
- Make events discoverable through a centralized catalog with taxonomies, enabling quick search, browsing, and access to their specifications similar to an API developer portal.
- Provide decentralized, self‑service access so developers can independently explore, register applications, and receive usage feedback—much like browsing a Netflix catalog.
- Implement decoupled, scalable access mechanisms that expose event topics without tightly coupling to underlying infrastructure, ensuring robustness and flexibility in larger architectures.
Full Transcript
# Scaling Kafka Event Endpoint Management **Source:** [https://www.youtube.com/watch?v=-vSdDHxKVYU](https://www.youtube.com/watch?v=-vSdDHxKVYU) **Duration:** 00:06:29 ## Summary - Use the AsyncAPI specification to formally describe Kafka event topics, giving them the same developer‑friendly interface and documentation standards as traditional APIs. - Make events discoverable through a centralized catalog with taxonomies, enabling quick search, browsing, and access to their specifications similar to an API developer portal. - Provide decentralized, self‑service access so developers can independently explore, register applications, and receive usage feedback—much like browsing a Netflix catalog. - Implement decoupled, scalable access mechanisms that expose event topics without tightly coupling to underlying infrastructure, ensuring robustness and flexibility in larger architectures. ## Sections - [00:00:00](https://www.youtube.com/watch?v=-vSdDHxKVYU&t=0s) **Managing and Scaling Kafka Events** - The speaker outlines how to describe, catalog, and discover Kafka event topics using AsyncAPI specifications to enable scalable event endpoint management. ## Full Transcript
have you been using kofka and been
really excited about the progress you
were making but wondering hey how can I
scale this usage out on a much larger
scale well if so you're going to love
our topic today on event endpoint
management hello my name is Jamil Spain
IBM developer Advocate and brand
technical specialist in the US Financial
Services Market to cover the topic of
event endpoint management I wanted to
bring out four major tips that you can
use use to which really encompasses the
discipline of describing and managing
your event topics here so we're going to
start off by filling these in and we'll
discuss these points as we go the first
is
that these event topics need to be
described now there's a specification
called the async API which is the pretty
much the industry standard on how to
describe events as topic or topics as
events um which is industry standard
here and what it really aims to do is
give uh events the same comfort as we
have grown to love with apis and let's
face it now um developers will want to
implement apis but they love to have
that kind of that Swagger interface that
they can quickly go ahead and and start
to go which brings me to my next Point
events should
be discoverable
and what this entails is uh kind of the
catalog format of having all the events
in one place tag with a taxonomy so it's
easy for me to search and and cataloged
and and very presentable for me to
actually go in and read the
specification for again another great
Trend that we're used to for a lot of
API developer portals the fourth we're
going to say is you should have
so I'll say that you'll have
decentralized access here so what this
pretty much says is people should be
free to explore all right they
Innovation usually comes um where I'm
I'm free to kind of go out and and use
it at my own Comfort I kind of
equivocate it to like when I'm on
Netflix and I get to browse the catalog
and really find movies as I want to go
kind of getting my own Rabbit Hole here
some of the trends that go along with
that is the capability to be
self-service where I get to register my
applications and truly have some way to
have trackable usage on what I'm using
getting that feedback there as well and
the last
is decoupled access so we really want to
find a way that's very scalable and less
fragile again the information or Kafka
as a whole the the the the process of
providing access to event topics should
not really impede on how I'm forced to
implement it here and so by putting this
interface in front that would really
give me a scalable way that I can expose
access to this and still accomplish what
I'm trying to do inside my architecture
here so hopefully that describes the
general Arena of managing events um um
and and the things you need to pay
attention to and watch out for let's
move over to the right and do a little
diagram about really depicting what it
looks like and how it works so for this
let's start out with our usual uh uh
diagram that everyone knows with this
how API management
works and again that we we already know
that apis get socialized let's say I am
a user and I'm sitting at my laptop and
I'm going to be looking at a developer
portal we'll call that acronym that DP
um and then there's the concept of a
Gateway all right API Gateway
and now as I browse in the developer
portal I see something that I want to
subscribe to Bally when I go to
implement I'm going to be actually going
consuming through the API Gateway and as
we know it has an interface to go out
and hit some other uh backend
Service as is so that's a a normal
practice that's that's always there
again all the apis are described in the
developer portal but this let's put
these the these particular uh
initiatives to to work here when event
import management comes into play let's
switch our marker here so with
events we're still going to have the
same developer portal
here a portal that describes and list
all our particular events as they go and
we're also going to have the concept of
a Gateway so that's called the EV the
event
Gateway all right so as I am still being
the user let's take the same user so
they're looking at apis and alongside
that they have access to also browse
topics here a third tier that they can
do which will then as they go to consume
these will go through the API Gateway
and then be able
to consume from from Kafka
there and provide the particular
information back as they would do so by
using the developer portal developers
get to quickly scale up and and and and
be able to um look at the events that
are there and know be able to test it
out quickly impit and get going and as
you become your Kafka become Kafka as a
service or a very uh Kafka first
environment where you want to expose a
lot of data directly through event
topics this is a great way with event
endpoint management to really get that
program going and facilitate that so
thank you for your time hope this was
very useful in your um u in your use of
Kafka here and I look forward to hearing
your stories about how you'll be using
event endpoint Management in your
architecture thank you if you have any
questions please drop us a line below
and if you want to see more videos like
this in the future please like And
subscribe