Azure Event Hubs for Apache Kafka | Azure Friday

>>Hey friends. Did you
know that Azure Event Hubs ingest more than two trillion
messages per day, and as a distributed
streaming platform Event Hubs enables you to stream your
data from any source, storing and processing
millions of events per second? Shupa is here to show us
how Event Hubs now works for Apache Kafka today
on Azure Friday. Hey friends. I’m Scott Hanselman and it’s another episode
of Azure Friday. I’m here with Shupa Jayasraki
and we’re talking about Event Hubs
and Apache Kafka. Two of my favorite
things in big data, and you’ve brought me
some cool demos as well.>>Yes. I’ll try.>>Yeah, what’s going on with Event Hubs as it relates
to Apache Kafka?>>Event Hubs as we know is a distributed streaming platform and Apache Kafka is also a distributed
streaming platform. We tried our best as Event
Hubs team to try to bring these two big data
streaming platforms together and that’s what
you’ll see and how it is.>>Cool. So you’re going to make it so they
can work together?>>Yes. If we start over, if you see what we see
with big organizations, is they have a lot of
data points that they want to try and analyze
so that they can improve, they can work on that, have better return of investment. In any enterprise lifetime, you will be building
data pipeline and if you see how a typical
data pipeline looks, it’s like you have producers or the data points that
you’re interested in. You start collecting,
staging and then ingressing, processing, modeling and doing. That’s the whole typical
data pipeline where Event Hubs or Kafka come in as the friend door for
any streaming platform, where you’re ingesting before you store your model
your process. It’s like a buffer and a temporal buffer for
your producers and consumers. You can process them and ingest and process
them to model answer.>>I see. So, my producers
can be anything. It could be IoT devices, it could be video
or click streams.>>Applications, click streams, web apps, sensor data.>>You said it’s like a buffer because some days you might have millions of requests a second and some
days you might not. So it’s coming and going.>>I call it a buffer because your producers might produce
at a different velocity and volume while consumers can consume at a different
velocity and volume. So, you want something in between for
your smart endpoint. The pipeline just stay
so you don’t have to, your producers need
not be bothered about when the producers are ready.>>There’s a whole ecosystem of ingestion systems here and
you’re unifying those.>>Yes. Exactly.
They’re just bringing that one big fat pipe so that your producers and consumers can have a straightened pipeline.>>Do you find
a lot of people out there that have streaming
data they might be using Kafka and they don’t know if they can
bring it to Azure?>>Yes, for sure. If you see the Conceptual
Architecture of Event Hubs, you have Event Hubs or
Kafka topic very similar, and the way Event Hubs or
Kafka has data is they have a partitions in
their architecture where data gets redistributed. So, you have a bunch
of even producers and a bunch of even receivers who are
interested in this data, and the data gets distributed
within Event Hubs or Kafka topics in
a uniform pattern. What happens with this, you’re having a huge downstream
parallelism where you are concurrently you’re
letting your receivers concurrently read from
all these partitions. It’s called a partition
consumer model not like the typical competing
consumer model that you would see
in an enterprise.>>It’s not necessarily
PubsHub either?>>You can make it as
a PubsHub because you have events or messages coming
in getting distributed, and couple of event receivers are interested in
only a couple of things. The thing is it’s a buffer
so it’s not a push model, it’s a pull model where you can do a lot on
your receiving end. What this gives you is a scale. If you have four receivers today and I want to add
six receivers, it just scales.>>Awesome. That’s
the great thing about Azure in general. It just scales.>>If you see, this
is how the Kafka is, that’s where many users
come and say, oh no I have Kafka, I have Event Hubs, both streaming platforms,
which one should I choose. Is it Event Hubs or Kafka. Because there are a lot
of advantages with Kafka. Kafka is open source. Apache Kafka is open source. It’s a software. It’s downloaded, and Event Hubs is on cloud. It’s a cloud offering. It’s a PaaS offering. So the couple of pain points
when we talk about Event Hubs or Apache Kafka
is which one do I choose? Many customers are
already on Apache Kafka. It has an on-premise story and Event Hubs
is only for cloud. That’s where we
thought we bring in this and say it’s
not ‘or’ it’s ‘and’. So, that’s what
our integration story is where we are trying to say we make-ably will
build with HTTPs and AMQP, but now we understand
Kafka protocol as well.>>Interesting. So this
reminds me of the time that I had the folks
from the Cosmos DB site, and they said Cosmos is amazing, it has all these features that you don’t see anywhere else. But it’s not something
that’s open source and it doesn’t have
a protocol people understand. So they put Mongo in front
of it and now everyone wins.>>That’s right.>>So the same thing
is happening here. You’re bringing Kafka to Event Hubs so that they work together, and people who have existing
producers can just go off.>>Yes. The biggest
advantage here is you’re not changing
any of your clients, you’re not changing your code, your producers,
your applications, your tools, your frameworks. Kafka offers framework
like connect which is trying to stream using Kafka with your source
and destinations, your data sources and
data destinations, which could be like
your file streams, your databases, your JDBC SAP. All these will just
work with Event Hubs.>>So, are you saying
that I could take an existing app that I had
written using Kafka libraries. Does it even know
Azure is a thing?>>Yes. It need not know
anything about Event Hubs.>>Okay. Do you have a demo?>>I do have. So, what I have here is
a sample Kafka producer. As you can see, there is
no hint of Event Hubs anywhere. It’s all using Kafka API, and the sample consumer as well, it’s a simple Java program. All I need to do on
my producer and consumer is add Bootstrap server entry, which is Event Hubs namespace. Now, how do you create
Event Hubs namespace? You can go to the portal. I have one already
created for you. This is testingfordemo
is my namespace. It’s a Event Hubs namespace. You can just do create and
click on “Event Hubs”.>>Okay.>>They even have
comes in and then you can start creating
your namespace with it. It’s just click, click,
click, and [inaudible].>>Do I have to
tell the namespace I want to use Kafka? Or do
I get that automatically?>>Every namespace is
Kafka enabled so you don’t have to explicitly
say that it’s Kafka.>>That’s nice.>>So, I already have
a namespace created.>>Okay.>>I’ll reuse that.>>I noticed in your
connection string, they’re using 9093, 9092, those are standard ports.>>Yes.>>So, really your app looks like your connection string looks like Kafka because it is Kafka.>>Yeah. Kafka,
most applications also use 9092 is an unsecured port, 9093 is a secured port. Event Hubs requires TLS handshake and 9093 makes a lot
of sense with that.>>Nice.>>So, yeah.>>Okay.>>I already have a testing for demo namespaces,
a sample namespace.>>Okay.>>What I need from is
these connection strings.>>Okay.>>This gives me the endpoint that is needed to
talk to Event Hubs.>>Right.>>So, this is where I can
get my connection string. I already have a
connection string pasted. This is where you need to
put your connection string, just copy and paste and
this is the namespace. What Event Hubs namespace
gives you is a unique FQD, a fully-qualified domain name and that’s what you
need to append as well along with the namespace name
and that’s where your bootstrap server or
your host server name would be.>>Okay.>>So, that done.
The same thing I would do for my consumer config as
well I’m just adding my bootstrap server name and my connection string as my endpoint and you’re all
ready to go, that’s it.>>Yeah, let’s do it.>>So, I have the Kafka producer.>>Okay.>>Okay. Let’s start. This is just sending some, say, 1,000 messages probably->>Okay.>>-to my namespace.>>So, you are doing a
[inaudible] and you’re running that over in
the producer side.>>Yes, it’s just simple
Java app that I’m running.>>Okay.>>So, it started.>>Here comes some data.>>Now, as I’m running that.>>Neither the producer
nor the consumer know it’s Event Hubs because
they’re talking to the Kafka protocol
on that board.>>Yes, absolutely.>>Okay.>>So, I started
my consumer app as well which will start receiving the events that
the producer sent.>>Okay. Look at that.>>Yeah, and if you see you can start
seeing the data here.>>That’s so nice, and you
see the data in the portal.>>Yes.>>In the comfortable
places you already know and you already
understand and I can see that you’ve got filtering and metrics you can
go and dig into.>>Yes, the metrics can-
Azure Event Hubs has metrics using Azure
monitoring which with which you can build
beautiful dashboards. You can use that. Kafka API also gives metrics
in its metrics that also can be used in lieu with Event
Hubs so mix and match works.>>Really. So,
the administrative tools are consumers as well and they
can go and view that. So, if you’ve already
built a Kafka dashboard you can point that at
Event Hubs as well?>>Yes, it should be because if it’s talking to Kafka API, you can talk to this as well.>>That’s so cool.>>It’s as simple as that.
So, we see that happening. While this is happening, I also want to show another
interesting thing. We said Event Hubs natively
supports AMQP and now we opened up for Kafka and
many customers are wondering, “Okay, that means if I
can send with AMQP and received with Kafka or send with Kafka and receive at AMQP? “.>>That’s clever.>>I say yes. So, I have
a sample sender here.>>C#. Okay.>>C#, I know. That’s
great. That’s my language.>>I know. This is
all our language.>>I know that’s
good school though.>>Yeah. So here, I don’t have any Kafka.>>Okay.>>I only have Event Hubs. This is the native Event Hubs API and I have as you can see.>>So, it says testing for demo. Is that the same namespace?>>Same namespace I’m
using for the [inaudible].>>Okay but there’s nothing
that says Kafka here. So, C# is thinking Event Hubs.>>Correct.>>Which Java think is Kafka.>>Correct. Although,
I have created another topic so that I can
show the demo distinctively. The previous one used
topic called test.>>Okay.>>This one is using
a topic called testeh.>>Okay.>>If you see our consumer app.>>Okay.>>Now, so it’s
consuming from testeh. The previous consumer was
consuming from the test topic. This will be using the testeh.>>Okay.>>So, let’s just run. The same thing goes.
Our Event Hubs client also requires an end-point.>>Right.>>Its connection string,
it’s the same thing. So, let me run this and it
will run it in the debug mode.>>Okay.>>They should start
sending the messages. [inaudible] Yes.
This is AMQP send.>>Okay.>>Let the messages
start and then start. Okay, it’s sending. Now, I will start the-.>>So,.NET back here
and Java in front.>>Java is.>>They’re taught in
two different universes, two different
libraries, all brought together by Event
Hubs using Kafka.>>Correct.>>That’s so cool. So,
there’s so many people out there who are probably watching this and thinking, “Oh, I already have something
that I can use it with this right away” and
no code changes just-.>>No code changes,
just a conflict change. Instead of putting
your bootstrap servers, you’ll be using the namespace
or the [inaudible] for Event Hubs and connection string which is an endpoint that
you need to talk to Kafka. Furthermore, things like there are tools like MirrorMaker which mirror your Kafka clusters be it on-prem or on cloud to
other Kafka cluster. So with this protocol, you can use tools like
MirrorMaker to mirror your on-prem Kafka clusters onto Event Hubs
which is on cloud. That’s like two line in configuration change and you’re migrating data from
on-prem to cloud.>>Wow. So, is this
available now? Can people start using this or?>>Yes, it’s available at GH now. It’s available in
all Azure regions.>>Okay.>>Yeah, you can start
using it. It stopped.>>All right.>>So, this is the last
of the slide that I have. So, when people say, “Okay, we integrated
Event Hubs with Kafka.” It’s now like why would I
need that and this slide here tries to explain everything how you get
a past experience, you’re not running your
zookeeper which is orchestrated to run or to
manage your Kafka clusters, there is no flavor of zookeeper. You are not running any clusters, you’re not running anything, you’re just streaming the data, thinking of scaling, and using
your current applications.>>Fantastic. All of these things you don’t have to think about. You get to go and
have that great Kafka ecosystem as a service.>>Exactly.>>Using Event Hubs.>>Yes.>>That’s fantastic.>>The other thing
is like you get not only the wider Kafka ecosystem
with Event Hubs, you get the wide Azure Event
Hubs ecosystem with Kafka. So, the mix and match works.>>Interesting.>>So, it can just work
with Stream Analytics, for the Azure functions, and you can use the storage, the blobs and Cosmos DB and everything that Azure
offers as well.>>Well, thank you so much
for sharing that with us.>>Yeah, thank you Scott.>>All right. I’m learning
all about Azure Event Hubs and Apache Kafka today
on Azure Friday.