A couple of weeks ago, I posted how I set up Kafka to run serverless in Azure by deploying Confluent Cloud.
If you have followed my blog lately, you have probably seen that I am interested in Azure Data Explorer and that I have a couple of conference talks coming. One being:
- How to do Real-Time Analytics Using Apache Kafka and Azure Data Explorer: We are looking at how to stream events from Apache Kafka to Azure Data Explorer (ADX) and perform user-facing analytics in near real-time.
The question is how to connect Kafka with ADX? You normally connect Kafka with another system using a Kafka Connect connector, and fortunately a connector exists for connecting Kafka with ADX: the Kafka Connect Kusto Sink Connector.
However, since I am running managed Kafka (Confluent Cloud, remember), I need a managed connector to run it in Confluent Cloud’s Kafka Connect. In the previous paragraph, I mentioned I was fortunate as we had a Kafka connector for ADX. Unfortunately, it is not a managed connector, so I cannot run it in Confluent Cloud - bummer!
So, this post looks at the various options we have if we want to use the Kafka Connect Kusto Sink Connector connecting Confluent Cloud in Azure with Azure Data Explorer. However, if you are not interested in neither Kafka nor ADX, the post may still be of use for you. The reason being it also covers running Docker images in Azure Container Instances (ACI).